Systems and methods for safety and business productivity

ABSTRACT

The present invention is a safety and business productivity system having the following components. One or more cameras capture video data having attribute data, the attribute data representing importance of the cameras. One or more video analytics devices process the video data from one or more of the cameras and detect primitive video events in the video data. A correlation engine correlates two or more primitive video events from the video analytics devices weighted by the attribute data of the cameras used to capture the video data. An alerting engine generates one or more alerts and performs one or more actions based on the correlation performed by the correlation engine.

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority from, U.S.Ser. No. 12/279,720, filed on Jul. 12, 2010 and issued as U.S. Pat. No.8,013,738 on Sep. 6, 2011, which itself is a national stage applicationof, and claims priority from, PCT Ser. No. PCT/US07/80488, filed on Oct.4, 2007, the entirety of both of which are hereby incorporated byreference herein.

FIELD OF THE INVENTION

The present invention is generally related to security and surveillancesystems. More specifically, this invention relates to an intelligentsecurity and surveillance system having alerts correlated using sensorydata from one or more sensors, the sensory data weighted by attributedata representing information about the source of the sensory data. Thepresent invention may be used to help fight crime and help ensure safetyprocedures are followed.

BACKGROUND OF THE INVENTION

As citizens of a dangerous world, we all face security and safety risks.Every day, 30 people die by gunshot in the U.S.—one every 48 minutes. Apolice officer dies from a gunshot wound every ten days. An intelligentsecurity and surveillance system may save lives.

Vandalism and damage to property decreases property values. One studyconducted by the London School of Economics found that “a one-tenthstandard deviation increase in the recorded density of incidents ofcriminal damage has a capitalized cost of just under 1% of propertyvalues, or £2,200 on the average Inner London property” (Steve Gibbons,The Costs of Urban Property Crime, 2003). An intelligent security andsurveillance system may prevent such vandalism.

Every year from 1996-2005, over a million motor vehicles were stolenevery year. That corresponds to one car stolen every 26 secondssomewhere in the United States. In 2004, the value of stolen motorvehicles was $7.6 billion and only 13% of thefts were cleared by arrests(Uniform Crime Reports, 2006). An intelligent security, surveillance,storage, and alerting system may help prevent stolen cars, and mayidentify stolen vehicles and hence aide in the apprehension of carthieves. Unfortunately, no existing surveillance system has theintelligence to correlate information about vehicles or has theconnectivity to national, international, state, or local law enforcementdatabases.

Violence in schools and on college campuses continues to rise, and hasincreased concern among students, parents, and teachers. A shooting atVirginia Tech University in 2007 resulted in the killing of 32 peopleand injured 24 others. In 2005, a professor at MIT was shot four timesin a parking lot on MIT's campus. In September 2007, two students wereshot by a fellow student at the Delaware State University. Shootings oncollege campuses are increasingly becoming a common concern. Anintelligent security and surveillance system on college campuses maythwart future shootings.

Therefore, as recognized by the present inventors, what are needed are amethod, apparatus, and system of alerting that weights input data fromdisparate systems to lower false alarm rates and to filter out unwanted,spurious, or intentionally distracting information.

It is against this background that various embodiments of the presentinvention were developed.

BRIEF SUMMARY OF THE INVENTION

One embodiment of the present invention is a video surveillance,storage, and alerting system having the following components. One ormore surveillance cameras capture video data having attribute data, theattribute data representing importance of the surveillance cameras. Oneor more video analytics devices process the video data from one or moreof the surveillance cameras and detect primitive video events in thevideo data. A network management module monitors network status of thesurveillance cameras, and the video analytics devices, and generatesnetwork events reflective of the network status of all subsystems. Acorrelation engine correlates two or more primitive video events fromthe video analytics devices weighted by the attribute data of thesurveillance cameras used to capture the video data, and network eventsfrom the network management module weighted by attribute data of devicecorresponding to the network event. An alerting engine generates one ormore alerts and performs one or more actions based on the correlationperformed by the correlation engine.

Another embodiment also includes a normalization engine for normalizingthe primitive events from the video analytics devices and the networkmanagement module.

Another embodiment also includes a privacy filter for filtering outprimitive events normalized by the normalization engine based on a setof privacy rules.

Another embodiment also includes a business filter for filtering outprimitive events normalized by the normalization engine based on a setof business rules.

Another embodiment also includes a compound event detection module fordetecting compound events composed of two or more primitive events.

Another embodiment also includes an event correlation module forcorrelating the primitive events and the compound events across time.

Another embodiment also includes an event correlation module forcorrelating the primitive events and the compound events across space.

Another embodiment also includes a rules engine for evaluating one ormore rules based on the correlation performed by the correlation engine.

Another embodiment also includes a learning engine for generating one ormore new rules based on the primitive events correlated by thecorrelation engine and the alerts generated by the alert engine.

Other embodiments of the present invention include the methodscorresponding to the systems above, the apparatus corresponding to thesystems above, and the methods of operation of the systems describedhere. Other features and advantages of the various embodiments of thepresent invention will be apparent from the following more particulardescription of embodiments of the invention as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system architecture for a video surveillance,storage, and alerting system according to one embodiment of the presentinvention;

FIG. 2 illustrates an architecture of a correlation engine according toone embodiment of the present invention;

FIG. 3 illustrates an architecture of a network management module inaccordance with one embodiment of the present invention;

FIG. 4 illustrates an architecture of a hierarchical storage manager inaccordance with one embodiment of the present invention;

FIG. 5 illustrates an architecture of a vehicle information module inaccordance with one embodiment of the present invention;

FIG. 6 illustrates an architecture of a video tip module in accordancewith one embodiment of the present invention;

FIG. 7 illustrates a topological map of a network generated by thenetwork management module in accordance with one embodiment of thepresent invention;

FIG. 8 illustrates a physical map of a network as monitored by thenetwork management module in accordance with another embodiment of thepresent invention, with FIG. 8A showing a street map view and FIG. 8Bshowing a satellite view;

FIG. 9 illustrates an interior map of a network as monitored by thenetwork management module in accordance with yet another embodiment ofthe present invention;

FIG. 10 illustrates a mathematical model of the present invention;

FIG. 11 illustrates a system architecture of another embodiment of thepresent invention;

FIG. 12 illustrates yet another system architecture of yet anotherembodiment of the present invention; and

FIG. 13 illustrates a flowchart of a process for video surveillance,storage, and alerting according to one embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a system, a method, and an apparatus forsurveillance, storage, and alerting. The present invention collects,stores, and correlates data from various sensory devices (such as videodata from video cameras), as well as meta-data about the collected data,and generates one or more intelligent alerts based on meta-data andattribute data of the devices used to detect the meta-data.

Definitions

As used herein, the term “meta-data” shall designate data about data.Examples of meta-data include primitive events, (including video andaudio events), compound events, meta-data extracted from video tips,network management events, and vehicle information. Meta-data alsoincludes compound events and correlated events, defined below. Meta-dataalso includes information added manually by a human reviewer, such as aperson who reviews a video tip, or a transcriber of a video speech.

As used herein, a “primitive event” is an atomic, indivisible event fromany subsystem. Primitive video events are events that have been detectedin the video, such as a people entering a designated area, vehicledriving the wrong way in a designated lane, or a package left behind ina given area. Primitive audio events include events that are detected inaudio data, such as gunshot events, a person screaming, glass breaking,etc. Meta-data extracted from video tips gives rise to video tip events.The network management module generates network events corresponding tonetwork occurrences, such as a camera losing network connection, astorage device going down, etc. Vehicle events are generated fromlicense plates detected on vehicles, and may include informationretrieved from one or more law enforcement databases. Legacy and othersystems also give rise to primitive events. For example, a card accesssystem generates a “swipe card detected” event with the correspondingunique card number whenever a card is swiped.

Primitive events may be generated automatically by various sensorydevices, or may be generated in software based on data from the sensorydevices. For example, a camera may generate an event corresponding tothe presence of a person. In another example, a gunshot detectioncomponent may generate a primitive event indicating that a gunshot wasdetected and the gunshot's estimated location. The primitive events areconfigurable by a system administrator. The system administrator maycustomize the types of primitive events that are activated and recorded.

In one embodiment, a human operator adds meta-data and thereby generatesprimitive events. For example, a human operator may add meta-dataindicating, “suspicious activity was observed at this location.”

As used herein, “compound events” shall include events that are composedof one or more primitive events. An example of a compound event istailgating. A tailgating event consists of a person entering adesignated area (primitive event) when no corresponding swipe/accesscard is detected (another primitive event).

As used herein, “correlated events” shall include primitive and/orcompound events that have been correlated across either space or time.An example of a correlated event is the same car (based on its licenseplate or vehicle properties) detected loitering in the same locationacross several days. Another example of a correlated event is the sameperson (based on his or her swipe card number) allowing tailgatingbehind him or her on more than one occasion.

As used herein, the term “attribute data” shall designate data aboutdevices or sources (such as sensory devices), such as the quality of thedata produced by the sensory device, the age of the sensory device, timesince the sensory device was last maintained, integrity of the sensorydevice, reliability of the sensory device, and so on. Attribute data hasassociated weights. For example, maintenance attribute data would have alower weight for a camera that was not maintained in the last 5 yearscompared to a camera that is regularly maintained every 6 months.Attribute data includes “attributes,” which are attributes of thesensory devices, and their associated “weights, or weight functions”which are probabilistic weights attached to data generated by thesensory devices. For example, an attribute would be “age of the device,”and an associated weight function would be a function decreasing withage. Some weights may also change with external events, such asmaintenance, time, and so on. For example, a weight associated with acamera may go down if the camera was not maintained for a period of timeand go back up after the camera was maintained. Attribute data may bedetermined by a system administrator, and/or determined heuristically.

In the case of video tips, attribute data refers to data about thesource of the video tips. For example, a video tip from an anonymoussubmitter will have different weights corresponding to the attributedata than a video tip submitted by a registered student with the studentusing his or her full name and ID number.

Attribute data is stored with the sensory data, and corresponds to theattribute data of the sensory device that captured the sensory data. Forexample, the quality of the camera (attribute data) that was used toacquire the video data is stored with the video data.

Meta-data (primitive events, compound events, correlated events, etc.)and attribute data are used throughout the present invention. Meta-datain the form of primitive events is used to detect compound events ofhigher value. Primitive and compound events are correlated across spaceand time to generate additional meta-data of even higher value. Theevents are weighted according to the attribute data corresponding to thesensory devices that generated the events. Primitive, compound, andcorrelated events may trigger one or more intelligent alerts to one ormore destinations. The meta-data is also used for forensic analysis tosearch and retrieve video data by event. Finally, meta-data andattribute data are both used for event correlation, for networkmanagement, and for hierarchical storage management of the video data.

System Architecture

One embodiment of the present invention is a system, a method, and anapparatus for video surveillance, storage, and alerting. FIG. 1 shows anexample of a system architecture 100 of one embodiment of the presentinvention. A network management module 101 monitors the health, status,and network connectivity of all components and subsystems of the system.(The dashed line represents the network management module monitoring theentire system.) The network management module monitors not only thedevices, such as the surveillance cameras, but also monitors thefunctional blocks such as the correlation engine for operation. Thenetwork management module generates network events reflective of thenetwork status of all subsystems. For example, the network managementmodule sends a network event indicating “connection lost to camera 1”when the network management module detects a network connection problemto camera 1. The network management module is described in greaterdetail with respect to FIG. 3.

Analogue surveillance camera 102 captures video data, which is digitizedby DVR 103. Video analytics device 104 detects primitive video events(“meta-data”) in the video data. The primitive video events, representedby line 140, may include such events as “person detected,” “vehicledetected,” etc., and are explained in detail below. Digital surveillancecamera 105 (which could be an IP camera) also captures video data. Videoanalytics device 106 detects primitive video events (“meta-data”) in thevideo data. Although only two surveillance cameras are shown, thepresent invention may be applied to any number and combination ofanalogue and digital surveillance cameras. The video analytics devicesmay consist of software running on a general purpose hardware device.Audio sensory devices 107 capture audio data, which is processed forprimitive audio events by audio analytics device 108. Examples ofprimitive audio events may include gunshot events, people screaming,glass breaking, etc. One or more additional sensory devices 109, such asa temperature probe (not shown), pressure probe (not shown), chemicalprobe (not shown), etc. provide sensory data that complements the videoand audio data.

A video tip module 110 receives “video tips” from one or more externalsources (which could be anonymous or non-anonymous, the externalssources are not shown in FIG. 1), extracts meta-data and attribute datafrom the video tips, and generates tip events based on the extractedmeta-data and attribute data. A “video tip” is a tip consisting of avideo clip, an audio clip, a still image, or other multimediainformation which can be submitted from a cell phone, or any portablecamera. Tips, that is, information from informants, are an importantsource of data. With the proliferation of video phones, tips are anincreasingly important source of information as multimedia informationis captured at the scene of a crime by well-meaning citizens and/orpolice officers. Video tips may be video clips recorded by video phones(cell phones with integrated cameras), digital cameras, handheld videocameras, etc. The video tip module is described in greater detail withreference to FIG. 6.

Numerous legacy systems, such as card access system 111, personnelsystem 112, etc. may be integrated into system 100 by the use of anappropriate normalization engine (to be described below). These legacysystems provide important “meta-data” events, such as “person A swipesinto building B,” etc. The legacy systems also provide importantinformation to the correlation engine, for example, “person A is aregistered student,” “person B is a faculty member,” etc.

Vehicle information module 113 retrieves information about a vehicledetected in the video data based on the detected vehicle's licenseplate, and generates vehicle events based on the information retrievedabout the vehicle. If a vehicle is detected in the video by videoanalytics device 104 or 106, vehicle information module 113 retrievesinformation about the vehicle from one or more law enforcement databases(not shown in FIG. 1) based on the detected vehicle's license platenumber. The vehicle information module is described in greater detail inrelation to FIG. 5.

A hierarchy of two or more data storage devices 130, 131, 132 stores thevideo data from the surveillance cameras, audio data from the audiosensory devices, data from other sensory devices, video tips from thevideo tip module, vehicle information, and data from other legacysystems. (The hierarchy of data storage devices is connected to thesurveillance cameras, the audio sensory devices, and the video tipmodule via a network.) A hierarchical storage manager (not shown)manages storage and cascade of the data among the storage devices. Thehierarchical storage manager is described in greater detail in relationto FIG. 4.

A normalization engine 114 receives primitive events such as primitiveevent 140, and normalizes the primitive events into a standardizedformat the system can recognize, identified as normalized event 115.Although one normalization engine is illustrated in FIG. 1 for clarity,in practice each type of sensory device may have its own normalizationengine. For example, there may be one normalization engine fornormalizing events from video analytics devices, another normalizationengine for normalizing events from audio analytics devices, anothernormalization engine for normalizing events from legacy systems such asthe card access system and the personnel system, etc. Alternatively, onenormalization engine as shown in FIG. 1 may have multiple modules foreach type of sensory device. The normalization engine(s) receivesinput(s) from the sensory device(s) and generates correspondingnormalized events for processing by the correlation engine. Anormalization engine is not necessary for those sensory devices thatproduce primitive events in the standardized system format. Normalizedevents 115 are placed in event queue 116 for processing by correlationengine 117.

Correlation engine 117 takes events from event queue 116 and performs aseries of correlations (across both space and time) on the events thatare described in greater detail below. After the events are picked offfrom the event queue 116 by the correlation engine, they are placed inpermanent storage in the events database 118 (an illustrative structureof this database is described below). The correlation engine 117 alsoqueries the events database 118 for historical events to perform thecorrelations described below. The correlation engine also receives inputfrom the configuration database 119 which stores configurationinformation such as device “attribute data,” rules, etc. The correlationengine 117 correlates two or more primitive events, combinations ofprimitive events and compound events, and combinations of compoundevents. Primitive events include primitive video events from the videoanalytics devices, audio events from the audio sensory devices, tipevents from the video tip module, network events from the networkmanagement module, or vehicle from events the vehicle informationmodule. The correlation engine is described in greater detail inrelation to FIG. 2.

Alert/action engine 121 generates one or more alerts and performs one ormore actions 124 based on the correlated events from the correlationengine. Examples of alerts include an email to a designated individual,an SMS message to a designated cell phone, an email to an Apple iPhone®or other multimedia-rich portable device, or an alert displayed on theoperator's interface 123. Examples of actions include “turn on lights,”“turn down thermostat,” etc. Detailed examples of possible actions thatmay be performed by the alert/action engine 121 are described in greaterdetail below. Alert/action engine 121 stores all alerts/actions thatwere performed in alerts database 122.

Cameras used in the present invention may be digital IP cameras, digitalPC cameras, web-cams, analog cameras, cameras attached to cameraservers, analog cameras attached to DVRs, etc. Any camera device iswithin the scope of the present invention, as long as the camera devicecan capture video. Some cameras may have an integrated microphone;alternatively, as shown in FIG. 1, separate microphones may be used tocapture audio data along with video data. As used herein, the terms“video,” “video data,” “video source,” etc. is meant to include videowithout audio, as well as video with interlaced audio (audiovisualinformation).

The system diagram shown in FIG. 1 is illustrative of only oneimplementation of the present invention. For example, the eventsdatabase and the video data may be stored on dedicated storage devices.Alternatively, a common server may house the events database and thevideo data.

Correlation Engine

FIG. 2 shows an architecture 200 of the correlation engine 117 accordingto one embodiment of the present invention. Primitive events 140 arereceived from one or more sensory devices, and are normalized into astandard format by the normalization engine 114 (which could be aseparate normalization engine for each device type). A privacy filter204 filters out primitive events based on a set of privacy rules. Theset of privacy rules are defined by a system administrator, and aredesigned to protect the privacy of individuals where the presentinvention is being used. The set of privacy rules instruct the systemwhich events to store, and which events to ignore. For example, in auniversity setting with a camera in a computer lab, a possible privacysetting may instruct the system to ignore all primitive events between 9AM and 5 PM. That is, the system would not record or process primitiveevents of people entering the computer lab during those hours. Inanother example in a university setting with swipe card access andassociated video, another privacy setting may instruct the system todisregard students swiping into their own dormitory during certainhours, but log and record students from other dormitories. (The policyfor recording video data is independently set from this privacy filter,so that video data may be recorded during those hours, but primitiveevents would not be stored or analyzed. Video data indexed by primitiveevents is more intrusive on privacy than merely recording un-indexedvideo data.) This privacy filter aims to strike a balance between, forexample, student safety and student privacy by disregarding eventsduring normal school hours or disregarding events of a certain type.Business filter 206 filters out primitive events based on a set ofbusiness rules. The set of business rules are defined by a systemadministrator, and are designed to customize the system to the businessprocesses in which the present invention is being used. The set ofbusiness rules instruct the system which events to store, and whichevents to ignore to align the present system with business processes.For example, in a corporate setting, a business rule would instruct thesystem to ignore all primitive events of a certain type (e.g., motion)in the data center during hours during which the data center isscheduled to be serviced. This business filter eliminates unnecessaryfalse alarms by disregarding events when they are not significant basedon normal business processes.

After the primitive events have been filtered by privacy filter 204 andbusiness filter 206, they are evaluated by compound event detectionmodule 208 for presence of compound events. An example of a compoundevent is “tailgating.” A tailgating compound event occurs when certainprimitive events are detected. That is, a tailgating compound eventoccurs when a single swipe card event from the legacy card access system111 is detected, while two or more people are detected entering thefacility on a camera that is directed at the entrance corresponding tothe swipe card's location. Compound events are defined by the systemadministrator as a combination of two or more primitive events. Compoundevents may include primitive events from one sensor, from two or moresensors, or even from two disparate types of sensors, as in thetailgating example above.

After compound events have been detected from primitive events, theprimitive and compound events are correlated across space by eventcorrelation module 210. Event correlation across space module 210 looksfor events occurring “substantially simultaneously” or in close timeproximity, across multiple sensors of varying types located acrossspace. Examples would include multiple tailgating events across afacility, or a loitering of two vehicles in different parts of a campus.Next, the primitive and compound events are correlated across time byevent correlation module 212. Event correlation across time module 212looks for historical event correlations between events detected now, andevents that occurred historically. Examples would include the sameperson (as identified by their swipe card) allowing tailgating onmultiple occurrences, the same vehicle (as identified by its licenseplate, or its make/model/color) loitering outside a college dormitory,or the same person (as identified by a log) stopped multiple times bythe security.

At each detection of a compound event by compound event detection module208, and each correlation across both space and time by eventcorrelation modules 210 and 212, the compound events and correlatedevents are stored in events database 118. Rule evaluation module 214evaluates a set of rules from rules database 216 based on the eventsstored in events database 118. Examples of event correlation and ruleevaluation are described in greater detail below.

Finally, alert/action engine 121 issues one or more alerts or performsone or more actions 123 based on the rules evaluated by the ruleevaluation module 214. The alerts/actions are stored in alerts database122. One of ordinary skill will recognize that the architecture shown inFIG. 2 is illustrative of but one correlation engine architecture and isnot intended to limit the scope of the correlation engine to theparticular architecture shown and described here. A more detailedmathematical explanation of the operation of one embodiment thecorrelation engine is described in greater detail below.

Network Management

FIG. 3 shows an architecture of the network management module 101according to one embodiment of the present invention. Network managementlayer 306 monitors the status of devices on the physical network 302 aswell as the status of applications 303, and keeps a record of device andapplication status in sources database 304. Network management layer 306detects all devices, including network cameras, servers, clientmachines, storage devices, etc. that are on the network. Topological mapmodule 308 generates a topological network diagram (an exampleillustrated in FIG. 7) of all networked devices. Physical map module310, which includes street map module 312 and satellite maps module 314,generates a physical map of the area being monitored. The physical mapmay be represented by a street map (as shown in FIG. 8A) or a satellitemap (as shown in FIG. 8B).

All surveillance cameras and audio sensory devices (such as gunshotdetectors) are displayed as icons on the physical map. “Plumes” (arcs ofcircles) are used to represent physical areas of coverage of thecameras, while “concentric circles” (or elipses) are used to representphysical areas of coverage of audio devices (such as gunshot detectors).The physical area of coverage for a surveillance camera is the physicalarea of the facility that is within the field of view of the camera.Since this value depends on resolution, as well as other cameraproperties (for example, a “fish-eye” camera has 180° of coverage),these values are obtained from the camera manufacturer and maintained asdevice “attribute data” (described below). Physical area of coverage fora gunshot detector is the physical area over which the gunshot devicecan accurately and reliably detect a gunshot. The physical area ofcoverage is obtained from the gunshot detector manufacturer andmaintained as device “attribute data” (described below). Typical gunshotdetectors have ranges on the order of approximately 0.25 to 1 mileradius, while typical cameras have ranges of several tens to hundreds offeet.

Finally, interior display module 316 displays interiors of buildings andshows devices and areas of coverage inside buildings. Interior displaymodule 316 is activated whenever an operator zooms into a building whilein either the street view or the satellite view. The interior displaymodule shows which interior portions of a building are covered (or notcovered) by the sensory devices, such as video cameras. Analogously tothe street view and the satellite view, the interior display shows iconsplaced on the floor plan corresponding to the locations of the camerasand plumes to represent areas of coverage of the surveillance cameras.(FIG. 9 shows an example of an interior display view.)

FIG. 7 shows an illustrative topological display as generated bytopological map module 308 of FIG. 3. The display shows an interface toview and manage topological display of all networked devices. Thedisplay shows IP addresses of all devices, as well as any other deviceinformation, such as MIB information obtained from SNMP agents thatreside on the devices. The icons also show the network status of alldevices (whether the device is connected, disconnected, awake, asleep,etc.). The icons blink, change color, or in some other way indicate adisconnected device or no signal to the device. The lines connecting thedevices to the backbone of the network may optionally show status of theinterconnections by displaying maximum (e.g., 100 MBs, 10 MBs, etc.) andcurrent bandwidth (whether busy, congested, free, etc.). The lines mayoptionally blink, change color, or otherwise indicate when there is nonetwork connectivity and/or bandwidth is insufficient for video streams.

The display automatically refreshes the view of the network and updatesthe display of the network. For example, if a camera is added, therefresh cycle automatically displays the new network with the newcamera. Any new devices plugged into the LAN are automatically displayedon the GUI. If an existing healthy device goes off-line, then its iconis represented in a different state (for example, a healthy device ingreen and an off-line device in red).

FIG. 8 shows an illustrative physical map display as generated byphysical map module 310 of FIG. 3. FIG. 8A shows an illustrative streetmap view as generated by street map module 312 of FIG. 3, while FIG. 8Bshows an illustrative satellite map view as generated by satellite mapmodule 314 of FIG. 3. The mapping data may be obtained from a mappingservice, such as Google Maps® or Microsoft Virtual Earth®.

The physical map provides a configuration interface to view and managephysical locations of all cameras, gunshot devices, other sensorydevices, storage devices, and any other devices and subsystems. Theinterface provides a mechanism to input locations of all cameras,gunshot detectors, other sensory devices, storage devices, and any otherdevices and subsystems of the network. A device is selected from thetopological map by clicking on the icon or selecting from a list.Physical locations of the device are selected on the physical map byclicking on the physical location, by entering the street address of thedevice, or by entering GPS co-ordinates (latitude and longitude) of thedevice. The physical locations of the device are saved in the sourcesdatabase 304.

Most mapping tools have good resolution up to the street or buildinglevel, but cannot zoom in past this level of detail. According to thepresent invention, finer detail may be shown on a floor plan, or a 3Dinterior map of the building. The floor plan view or 3D interior map isautomatically displayed when an operator attempts to zoom into aparticular building. For example, a bitmap of the building floor planmay be displayed to show camera locations inside a building when a userclicks on the building. As described previously, the interior displaymodule 316 of FIG. 3 generates and controls the interior map. FIG. 9shows an illustrative floor map as generated by interior display module316. The present invention is not limited to interior display in a floormap view as shown here. The interior may also be displayed in a 3D map(not shown), or another alternative representation of the interior of abuilding.

Heirarchical Storage Manager

During daily operation of the present invention, large amounts of dataare generated. For example, a typical 3 Megapixel digital surveillancecamera generates images of approximately 280 Kbytes per frame. If thiscamera were running at 5 frames per second, it would generateapproximately 60 GB per day. If an organization wanted to archive thedata for one month, it would take approximately 1.8 TB, and if theorganization wanted to archive the data for one year, it would takeapproximately 22 TB. In a typical application having 100 surveillancecameras around a particular facility, this translates into approximately6 TB per day, or approximately 180 TB per month, or over approximately2,000 TB per year! Ideally, requested data should be retrieved at thefastest rate and this is possible only if all of the data is availableon high-speed devices at all the time, but this is beyond the ability ofmost organizations. The Hierarchical Storage Manger (HSM) plays animportant role in providing large amounts of permanent data storage in acost-effective manner. That is, data files which are frequently used arestored on higher cost storage medium (like cache discs) but areeventually migrated to lower cost storage medium (like tapes ornetworked storage) if the data files are not used for a certain periodof time (or as per the defined migration policy). When a user requests adata file, which is on a slower storage medium (such as tape), it isautomatically made available, and is moved to a faster storage medium ifit is frequently accessed by the user.

The main benefits of the Hierarchical Storage Manager include thefollowing: 1) Support for rule-based migration and archive policy—oncethe rules and policies have been defined, HSM manages everythingautomatically. Cascading of data from higher storage medium to lowerstorage medium and vice-versa is automated based on policies defined. 2)Based on inputs provided by the system, HSM builds its own rules andpolicies—inputs can include storage limit threshold values (e.g., whenthe down-cascading has to be performed). 3) HSM reduces the totalstorage cost as data accessed less frequently resides on lower coststorage. 4) The performance is improved as unused data is moved to lowerlevel storage devices and frees up higher level (faster) storagedevices, thus increasing overall system performance) 5. HSM cutsadministrator time by not requiring manual data archiving, deletion tofree up disk-space, and manual data retrieval. 6) Disaster management issupported by automatic online data backups. 7) Data is automaticallycascaded up when the system accesses data. 8) The total amount of storeddata can be much larger than the capacity of the disk storage available,since rarely-used files are cascaded down to low-cost storage media.

The storage hierarchy may include hard disk, optical disk, magneticdisk, flash memory, tape memory, RAID array, NAS (Network AttachedStorage), SAN (Storage Area Network), or any other physical or virtualstorage media. An illustrative data storage hierarchy used by the HSMmodule may be:

1. Local RAID array (magnetic hard disk)

2. Networked disk array (SAN, NAS, etc.)

3. Tape array (e.g., Automated tape library)

4. Tape stored on the shelf after tape array is full

The following example is directed to video data, but the principles ofthe present invention may be applied equally to other data beingprocessed by the system, including audio data, video tips, as well asother data and related meta-data. Therefore, the use of the term “videodata” is not intended to limit the application of the HSM module to onlyvideo data, and is used illustratively only.

Video data may be cascaded down the storage hierarchy based on itsimportance (Y). The importance (Y) may be calculated as a weightedaverage of the attributes of the video data (including attributes of thedevice used to capture the video data). Examples of attributes of thevideo data include, but are not limited to, the following:

-   -   1. Resolution of the video data (R)    -   2. Age of the camera used to capture the video data (A)    -   3. Time since last maintenance of the camera used to capture the        video data (TM)    -   4. Location of the camera used to capture the video data (L)    -   5. Reliability of the source of the video data (whether it's a        camera, anonymous video tip, etc.) (RS)    -   6. Time since the video data was last accessed (TS) (TS=time        since data was stored if data has not been accessed yet)    -   7. Events detected in the video data (people detected, motion        detected, etc.)    -   8. Time period the video data was recorded (e.g., if monitoring        safety in a data center, then a period when the data center is        empty or during non-working hours has lower importance)

Importance of the video data (Y) is used to cascade the video data, andmay be calculated as a weighted average, as shown in Equation A.

$\begin{matrix}{Y = {\sum\limits_{i = 1}^{i = N}\;{w_{i} \cdot a_{i}}}} & (A)\end{matrix}$

where Y=importance of the data, a_(i)=attributes of the data (Σa_(i)=1),w_(i)=relative weights of the attributes (Σw_(i)=1), and N=total numberof attributes.

If  t₀ ≤ Y ≤ 1  then  data  is  stored  in  highest  (first)  hierarchy.If  t₁ ≤ Y < t₀  then  data  is  stored  in  second  hierarchy.If  t₂ ≤ Y < t₁  then  data  is  stored  in  thrid  hierarchy.…If  0 ≤ Y < t_(n)  then  data  is  stored  in  lowest  (last)  hierarchy, where  1 > t₀ > t₁ > t₂ > … > t_(n) > 0.

For example, in a case of six attributes each weighted equally, theimportance Y may be calculated as shown in Equation B:Y=(L+R+A+RS+TM+TS)/6  (B)

The preceding sample equations used to calculate the importance (Y) ofvideo data are illustrative of but numerous such expressions, and arenot intended to limit the scope of the present invention to theequations and terms shown here. Other attributes of the video data maybe used to determine the importance of the video data. In addition,fewer than the attributes listed here may be used to determine theimportance of the video data. Finally, an alternative expression otherthan a weighted average, such as a non-linear equation, may be used todetermine the importance of video data from its attributes.

The video data is divided into segments. Segments may be measured indays, hours, minutes, or seconds. The system administrator selects thesegment length, and the segment length determines the minimum atomicunits of video data that the HSM module cascades. Each segment of videodata has an associated entry in an internal HSM database. The internalHSM database keeps track of the importance of each segment of videodata, and its location in the storage hierarchy, as well as its actuallocation within each hierarchy. An entry is stored in the internal HSMdatabase describing the importance of each segment of the video data foreach device. As illustrated in equation (A), if importance of a videosegment is less than T (where T is defined by the system administrator),then that segment of video data is cascaded down to the next level. Forexample, if video data has an event (as recorded in the events table)and has been accessed frequently, then it has a higher importance (Y)than video data without any events. All events may not be weightedequally in determining importance (Y). If video data has an event ofhigh importance (as recorded in the events table, such as a gunshot),then this video data has higher importance.

When a given hierarchical level becomes near full, the video segments oflowest importance are automatically cascaded to free space for new data.

For example, importance may be a function of the time since the data waslast accessed. The data stored is evaluated on the basis of the age ofthe data, for example, if the data is more than X days old (where X isset by the administrator) and otherwise has no other attributesassociated with it, and has not been accessed, then this data iscascaded to the next level of storage devices.

For example, if the video data has no primitive events detected, but hasbeen accessed frequently, then this data will remain on the disk until Xdays (X is set by the administrator) of the last access time. If thevideo data has primitive events, but has not been accessed at all, thenthis data will be cascaded to the next level storage device after Y days(Y is set by the administrator) of the date of storage. If the videodata has primitive events and has been accessed continuously, then thisdata will remain on the disk until an administrator manually forces acascade from the disk.

In one embodiment, location of the camera used to capture video data isone factor in calculating importance of the video data. For example, ifthe location of a camera has high importance (for example, the datacenter), video from the camera will have higher importance and will bestored for a specified longer period.

In one embodiment, video data retained after the normal cascade isalways (1) some amount of time before the event and (2) some amount oftime after the occurrence of the event (these values are set by theadministrator). For example, 5 minutes of video before an event, and 5minutes of video after the event, are always retained along with theevent.

In one embodiment, all data has an attribute that reflects when data waslast accessed. Data that is recently accessed is likely to be accessedagain, and thus its importance will be higher and it will not be movedto a lower hierarchy. This enables operators to retrieve data that hasbeen recently accessed with lower delay.

FIG. 4 shows an architecture 400 for a hierarchical storage manager 401which is used to manage the storage of video data as well as other dataon the n-tiered storage configuration shown in FIG. 1 as storage devices130, 131, and 132. (The hierarchical storage manager is not shown inFIG. 1.) Video data from cameras 102 and 105, as well as other data fromother sensory devices, enters the hierarchical storage manager asdata-in line 402. An API interface 408 provides a common interface tostore data to, and retrieve data from, the hierarchy of storage devices.The API interface provides a standardized set of function calls whenstoring data, as well as when retrieving data. Examples of interfacecalls are shown in equations (C) and (D):StoreData(pointer to video data, Camera ID, Time)  (C)pointer ReadData(CameraID, Time)  (D)

In equation (C), the function StoreData stores the data referenced bypointer video data and corresponding to camera identified by CameraIDand time identified by Time into the storage hierarchy. In equation (D),the function ReadData returns a pointer to video data corresponding tocamera identified by CameraID and time identified by Time.

When storing data, the HSM rule module 410 determines on which storagedevice video data and other data should be stored based on events storedin event database 118, and configuration information (“attribute data”)stored in sources database 304. The HSM rule module 410 then stores thelocation information corresponding to the location of the stored data inan internal database, the video management database 414. When readingdata, the HSM rule module 410 determines on which storage device thedata is stored by checking the video management database 414.

The HSM rule update module 416 updates the video management database 414based on requested video data. For example, video data and other datathat is more frequently accessed are moved to faster storage devices.The HSM storage/retrieval module 418 manages the actual storage andretrieval of data. The HSM storage/retrieval module 418 interfaces withRAID controller 420 to access video data from a RAID array consisting ofdisks 426, 428, and 430. Three disks are shown for illustrativepurposes, but any number of disks is supported by the present invention.The HSM storage/retrieval module 418 interfaces with Tape controller 422to access video data from a tape array consisting of tape drives 432 and434. The tape controller may also interface to an Automatic Tape Libraryconsisting of hundreds of tapes automatically managed by a robotic arm.Finally, HSM storage/retrieval module 418 may also interface withNetwork Interface Controller (NIC) 424 to access video data via network(such as the Internet) 436 from remote, network-attached disks, such asSAN (Storage Area Network) 438 or NAS (Network Attached Storage) 440.Two networked disks are shown for illustrative purposes, but any numberof networked disks is supported by the present invention.

In one embodiment of the present invention, video data is captured andbacked up continuously to a remote location. The video data may be sentvia a network, such as the Internet, or a dedicated fiber optic line, toa remote, secure location. If the local copy of the data is damaged,destroyed, or tampered with, the copy in the remote location may beaccessed and analyzed. All video data may be automatically archived tothe remote location.

In one embodiment of the present invention, storage media 438 and 440serve as continuous live backup of the video data and are connected bytransmission media 436. Transmission media 436 may be a dedicated fiberoptic line or a public network such as the Internet. Storage media 438and 440 may be hard disk, magnetic tape, and the like.

The HSM Module provides centralized storage management operations withdata migration, archiving and restoring while reducing complexity andmanagement costs. HSM protects against data loss and other failures bystoring backup, by efficient space management for data, as well ascompliance and disaster recovery of data in a hierarchy of off-linestorage. An intelligent data move-and-restore technique andcomprehensive rule-based policy automation work together to increasedata protection and potentially decrease time and administration costs.

In one embodiment, in order to preserve user data in case of hardwarefailure or accidental removal, files written into an HSM-managed filesystem are backed up continuously to an offsite location. All data isimmediately compressed and backed up as soon as it is recorded to aback-up device. The back-up device is online and is always a second copyfor online data. Data and backed-up data are always synchronized. Apolicy could be defined to force the existence of a backup of a filebefore the file can be migrated from a higher level to a lower level.

In order to maximize the efficiency of data management, fresh data isstored on a cache drive, which is usually a magnetic hard disk. Oncedata meets a predefined rule, policy, or a threshold value based on itsimportance (Y) as defined above, data is moved from the high-coststorage medium to a lower-cost storage medium and gradually to tapes.HSM performs these functions automatically. A system administrator canconfigure the rules, specify the policies, or set the threshold valuesfor the HSM. Based on these rules and policies, migration and archivingare triggered. The rules may also be defined to move specific files,purge files, or to define the number of files to move at any one time.

The essential difference between migration and archiving is thebi-directional interface for dynamic retrieval provided by migration.Dynamic retrieval occurs when restoring the data back to diskautomatically when it is accessed and made available for processingagain. The ability to transfer data across the disk and tape interfacein both directions is system controlled (that is, automatic). That is,migration moves data from higher cost storage medium to the immediatenext low cost storage medium. Archiving, on the other hand, moves thedata permanently to tapes that may be shelved away for intermittentaccess.

HSM Migration: Migration physically moves selected data to differentauxiliary storage pools. It moves data from fast, high performance diskto slower or compressed disk, networked disk, automatic tape library, orsome other slower storage pool. This results in saving space on the fastdisk. Except for possible changes in access times, data that has beenmigrated is still fully available to any application that was able toaccess it before the data was migrated. Now the data will be accessiblefrom the migrated area. If required, data will be moved from slow diskto fast disk. Migration operations are performed automatically based ondefault rules. (The administrator can override rules so that data may bemigrated as required). Policies are defined for data migration.Migration of data is done at a predefined level (e.g., migration is maybe done at a camera- or folder-level, but not at file-level) to maintainapplication transparency. Migration of data is also done at a predefinedtime interval (e.g., every minute of data is always processed togetheras one segment). Rules can also be defined for avoiding migration ofspecific files. For example, a segment of video data that has an eventwill be stored longer on higher cost storage medium than a segment withno events. As explained earlier, migration may be based on theimportance of data, and a sample calculation of importance was shown inEquations (A) and (B). The equation by which importance of video data iscalculated is not predetermined, and may be customized by the systemadministrator. The migration criteria are given as inputs to the HSMmodule.

An illustrative migration process includes the following steps:

-   -   1. Identify segments of data suitable for migration    -   2. Establish suitable migration criteria based on the        importance (Y) of the video data. Criteria are the rules,        weights, or policies to determine which data qualifies for        migration.    -   3. Establish a migration policy, which triggers data migration        from the high-cost storage to low-cost storage and vice versa,        such as nightly, weekly, monthly, when the disk is 90% full,        etc.    -   4. Add the migration jobs to the scheduler        HSM Archiving: Archiving creates an interface from disk to        “shelved” tape allowing moving of inactive data to a less        expensive form of storage. Archiving selects infrequently used        segments of video data, saves them to tape, and then deletes        them from disk. This action frees up storage space. Archiving of        data to tape saves disk space on primary (fast) disk because it        moves the data to a less expensive form of storage. The HSM        Module keeps track of information about the segments of video        data that are archived. When a segment of video data is        recalled, the tape must be retrieved from the shelf and the file        is restored to the disk. Threshold values or policies based on        the importance (Y) of the video data are defined by the        administrator to start archiving. Archiving of data is done at        specified levels (e.g., archiving is done at folder-level or        camera-level, but not at file-level).

Sample archiving process includes the following steps:

-   -   1. Identify segments of data suitable for archiving    -   2. Establish suitable archive criteria based on the        importance (Y) of the video data    -   3. Establish an archive policy, which triggers data archiving to        tapes, such as weekly, monthly, when the disk is 90% full, etc.    -   4. Establish a media policy. The media policy consists of        information about the tape media, which are inputs to the        archiving policy, and also prevents erroneous use of active        media prior to expiration.    -   5. Archive logs        HSM Rules Engine: When a system state matches the predefined        policy, the appropriate migration or archiving action is        triggered. A system state such as a disc capacity crossing a        threshold value may trigger cascading of data to the next level        in the hierarchy. An HSM internal database for managing data on        the storage medium is maintained for all the data that is        stored, migrated and archived. This information keeps track of        the data's location, archive status, frequency of use and any        other attributes that are relevant to the HSM Module.        HSM Audit Trails: Audit trails are maintained by the HSM Module.        Data privacy is a major cause of concern, and the audit trail        keeps track of who has accessed each segment of video data and        when. The audit trail includes information about which data        segments were accessed, the type of data accessed, time at which        the data was accessed, by whom the data was accessed, and other        parameters. Each time anyone accesses a video segment from the        HSM Module, audit information is stored in an audit database.

Some sample HSM user roles which may be used for HSM auditing purposesinclude the following:

-   -   1. Administrator    -   2. Forensic analyst(s)    -   3. Operators    -   4. Management    -   5. Other authorized personnel (customer-specific)    -   6. Correlation engine (This is an application-internal user        which is going to access the data most frequently and perhaps        continuously. For example, when a suspicious vehicle is detected        in a parking lot and the correlation engine has a predefined        rule to search for all other instances where this camera        detected the same vehicle, and where this data was not        previously stored as meta-data.)

In one embodiment, the HSM Module may provide seamless compression andencryption services for data on the fly.

Vehicle Information Module

FIG. 5 shows an architecture 500 of one embodiment of the presentinvention in which the vehicle information module is used to retrieveinformation about vehicles detected in the video data. Video data fromcamera 105 is processed by license plate recognition module 502 toextract a license plate string and a state of the license plate (e.g.,Florida, Michigan, etc.). In cases in which multiple hypotheses arereturned by the license plate recognition module (such as 01234 andO1234), both possible results are queried. License plate strings andstates extracted by the license plate recognition module 502 are inputinto the vehicle information module 113, which queries one or more lawenforcement databases 506 with the license plate and state as searchstrings. Numerous law enforcement databases are envisioned to be withinthe scope of the present invention, including warrants database 508,wanted persons database 510, stolen plates database 512, mug shotdatabase 514, and any other law enforcement database that may beavailable, including FBI, Interpol, state and local databases. Thevehicle information module first queries a property record database 507for the license plates detected by the license plate recognition module502, to determined a registered owner of the vehicle. The vehicleinformation module then queries the other law enforcement databases,such as the warrants database 508, the wanted person database 510, orthe mug shot database 514 for the registered owner based on the resultsfrom the query to the property record database 507. The vehicleinformation module 113 may also query certain databases directly withthe recognized license plate, such as the stolen plates/stolen carsdatabase 512.

The present invention may query FBI, Interpol, state, and localdatabases. The present invention may query police, sheriff, and otherlaw enforcement databases. The present invention may query for recentcrimes, related arrests, outstanding or historical warrants, and pastconvictions. The present invention may query the FBI Most Wanted, aswell as Interpol Wanted Fugitives list.

After any relevant information is retrieved from the law enforcementdatabase(s) 506, the information is passed to a vehicle informationnormalization engine 516, which may be a component of normalizationengine 114 of FIG. 1, which translates the vehicle information intoappropriately formatted events that the correlation engine 117 canprocess. The vehicle events are stored in events database 118, and fedto correlation engine 117. Correlation engine 117 then performsfiltration, compound event detection, space-and-time event correlation,and rule evaluation as described in greater detail in relation to FIG.2, and stores any results in events database 118. Finally, as previouslydescribed, alerts/action engine 121 generates one or more alerts and/ortriggers one or more actions 124 based on triggers from the correlationengine, and stores the generated alerts/action in alerts/action database122.

Video Tips

FIG. 6 shows an architecture 600 of one embodiment of the presentinvention adapted to receive “video tips” from external sources. A“video tip” is a tip which includes a video, an image, a soundrecording, or any other multimedia recording, whether taken by a citizenor from some other source. In the context of the present invention, a“video tip” shall include any tip that has multimedia content, whetherit is a still image, a video, audio, or any other multimediainformation. In the context of the present invention, any such tip willbe within the scope of the phrase “video tip.” Video tips may be takenby video phones (cell phones with integrated video cameras), portablecameras, video cameras, etc. Video tips may be submitted via emaildirectly from a cellular phone, via MMS (Multimedia Messaging Service),or first uploaded to a computer and then emailed or uploaded to aserver. Increasingly vigilant citizens, as well as police officers withportable cameras, are capturing video information that could beimportant to solve and prevent crimes. (Before the present invention,“video tip” information is not archived, indexed, or maintained in amanner that is conducive to intelligent, pro-active alerting, orretrospective and forensic analysis.)

Video tips may be submitted from camera phones 602, 603 (phones withintegrated cameras), smart phone 604 (such as Blackberry®, Windows®Mobile phones, PocketPCs, or any smart phone with integrated cameras),or multimedia phone 606 (such as Apple iPhone® or other multimediaphone). Video tips may also be captured by a portable video camera 607,a portable still camera (not shown), a portable microphone (not shown),and in general any portable recording device which may or may not beInternet-enable. The portable video camera 607 (or other portabledevice) may be connected to personal computer 608 (or any otherInternet-enabled device), and the “video tip,” including any meta-datasubmitted by the tipster, may be uploaded via the computer 608. Thevideo tip may be submitted via a user interface, such as a web interfaceon a public (Internet) or private (Intranet) website. (For example, aperson would log into the system via the Internet and upload a video ofa crime that the person caught on video.)

An organization may setup a tip email address such as tips@sju.edu,and/or MMS address (Multimedia Messaging Service, an extension toSMS—Short Messaging Service, which is text-only), such as (617)455-TIPSto receive the video tips. The video tips are transmitted via theInternet 610, or any other local or global network, to mail or MMSserver 612, which runs a mail server or MMS server application, whichreceives submitted video tips.

Video tip normalization engine 614 processes the video tips received bythe mail/MMS server 612. Video tip download module 616 periodically (forexample, every 30 seconds) polls the mail/MMS server 612 and downloadsany newly received video tips. Video tip storage module 618 stores newlyreceived video tips into the hierarchical file system, via HSM manager401 which manages a set of storage pools 426, 432, 438, as was describedpreviously. Meta-data/attribute data extraction module 620 extractsmeta-data from the downloaded video tip. Examples of extracted meta-datainclude sender's email address (if sent via email), phone number (ifsent via MMS), location (if available), IP address (if uploaded viacomputer 608), date and time sent, and any meta-data in the form ofcomments submitted by the tipster. Attribute data is also assigned tothe video tip by the meta-data/attribute data extraction module 620based on such factors as the identity of the informant, the quality ofthe video, the reliability of the source (e.g., whether anonymous or aregistered student), other tips that are entering the systemcontemporaneously, etc.

After the video tip has been received and automatically processed, itscontent and extracted meta-data are presented to a reviewer for furtheranalysis and comment. The reviewer may enter additional meta-data fromoperator interface 623 via additional meta-data module 622. Thetipster's as well as the reviewer's meta-data is stored in video tipmeta-data database 625 via meta-data storage module 624. Finally, a tipevent is generated by the video tip event generation module 626corresponding to the extracted meta-data and attribute data, and storedin event queue 116. The correlation engine processes the tip event fromthe event queue 116 as previously described in relation to FIG. 1.

Database Design

The following tables and associated description shows illustrativedatabase schemas that may be used in an implementation of the presentinvention. It is to be understood that these schemas are illustrative ofbut one manner in which the present invention may be practiced, and thepresent invention is not limited to the particular database designsshown and described here.

Seven core database schemas will be shown and described. The meta-dataparameters table (Table 1) describes the various primitive and compoundevents that are detected and recorded by the present system and theirassociated parameters. The meta-data types table (Table 2) defines theprimitive event types that may be detected and recorded, defines thecomposition of compound events, and assigns absolute values (used by thecorrelation engine) to the meta-data types. The events table (Table 3)is an important database used by the correlation engine, and stores theactual primitive and compound events that were detected, as well as anindex into the corresponding video data. The sources table (Table 4)defines the various devices (including sensory devices), and theirassociated attributes and weights, and is the core database used by thecorrelation engine, network management module, and the HSM module. Therules table (Table 5) defines the rules defining the alerts and alertconditions used by the alert/action engine. Finally, the video tipmeta-data table (Table 6) and the license plate meta-data table (Table7) stores the meta-data associated with the video tips and the detectedlicense plates, respectively.

TABLE 1 Meta-data parameters table MDParametersID Nickname MDTypeIDSrcID MD_TimeStart MD_TimeEnd  6 Motion in Camera 1  1 1 17:00   8:00 .. . . . . . . . . . . . . . . . . 10 Person Enters Server Room 23 4 0:0023:59 11 Swipe Card Detected to Server Room 22 9 0:00 23:59 12Tailgating 24 4 0:00 23:59 13 Anonymous Video Tip 98 22 0:00 23:59 14Registered Student Video Tip 98 23 0:00 23:59 15 Stolen Plate 99 217:00   8:00 16 Camera 1 loses connection 105  1 0:00 23:59

Table 1 shows a sample meta-data parameters table, which stores thevarious primitive and compound events that are detected and recorded bythe present system and their associated parameters. “MDParametersID” isa primary key that uniquely identifies the meta-data parameter,“Nickname” defines a short phrase that describes the event, “MDTypeID”is a foreign key into the Meta-data types table (Table 2) that definesthe type of event, and “SrcID” is a foreign key into the Sources table(Table 4) corresponding to the device that detects this particularevent. Finally, “MD_TimeStart” and “MD_TimeEnd” are privacy or businessfilters that define the times during which the particular event isactive.

For example, the row “MDParametersID=6” corresponds to an event with anickname “Motion in Camera 1.” This event has “MDTypeID=1”, which byexamining Table 2 corresponds to a motion event. It has “SrcID=1”, whichby examining Table 4 corresponds to Camera 1 located in a lobby. Basedon “MD_TimeStart” and “MD_TimeEnd”, this event is only being monitoredand recorded between the hours of 5:00 PM (17:00) and 8:00 AM (8:00) toprotect privacy or to follow a business rule.

The row “MDParametersID=10” corresponds to an event with a nickname“Person Enters Server Room.” This event has “MDTypeID=23”, which byexamining Table 2 corresponds to the detection of a person. It has“SrcID=4”, which by examining Table 4 corresponds to Camera 34 locatedin a server room. This event is always being monitored and recorded(0:00 to 23:59).

The row “MDParametersID=11” corresponds to an event with a nickname“Swipe Card Detected to Server Room.” This event has “MDTypeID=22”,which by examining Table 2 corresponds to a swipe card. It has“SrcID=9”, which by examining Table 4 corresponds to a swipe card readerin the server room. This event is always being monitored and recorded(0:00 to 23:59).

The row “MDParametersID=12” corresponds to an event with a nickname“Tailgating.” This event has “MDTypeID=24” which by examining Table 2corresponds to a compound event called tailgating. It has “SrcID=4”corresponding to the server room. This event is always being monitoredand recorded (0:00 to 23:59).

The row “MDParametersID=13” corresponds to an event with a nickname“Anonymous Video Tip.” This event has “MDTypeID=98” which by examiningTable 2 corresponds to a video tip. This event has “SrcID=22” which byexamining Table 4 corresponds to an anonymous source of video tips. Thisevent is always being monitored and recorded (0:00 to 23:59).

The row “MDParametersID=14” corresponds to an event with a nickname“Registered Student Video Tip.” This event has “MDTypeID=98” which alsocorresponds to a video tip. This event has “SrcID=23” which by examiningTable 4 corresponds to a registered student being a source of videotips. This event is always being monitored and recorded (0:00 to 23:59).

The row “MDParametersID=15” corresponds to an event with a nickname“stolen plate.” This event has “MDTypeID=99” corresponding to a stolenplate event type. This event has “SrcID=2” which corresponds to a camerain an entrance to a parking lot (not shown in the sources Table 4). Thisevent is always being monitored and recorded (0:00 to 23:59).

The row “MDParametersID=16” corresponds to an event with a nickname“Camera 1 loses connection.” This event has “MDTypeID=105” correspondingto a network event. This event has “SrcID=1” which corresponds to Camera1 located in the lobby.

TABLE 2 Meta-data types table MDTypeID Description AbsVal CompoundEventTimeFrame 1 Motion 3 null null . . . . . . . . . . . . . . . 22 SwipeCard −1 null null Read 23 Person 1 null null Detected 24 Tailgating 5 23<AND 0:10 NOT> 22 . . . . . . . . . . . . . . . 98 Video Tip 6 null null99 Stolen Plate 50 null null 105 Network Event 60 null null

Table 2 shows the meta-data types table, which defines the primitive andcompound event types, and their associated absolute values. “MDTypeID”is a primary key that unique identifies the type of event, and“Description” provides a short description of the event type. “AbsVal”defines the default absolute value that is associated with thatparticular event type. The absolute value is used by the correlationengine to assign absolute values (x_(i) and v_(i) in Equations 20-22below) to various types of events, before they are weighted by theattribute data (w_(i) in Equations 20-22 below). “CompoundEvent” definesthe relationship between compound and primitive events, and “TimeFrame”defines the period of time during which two primitive events must occurinto order to be eligible for detection as one compound event. “CompoundEvent” and “TimeFrame” are null for primitive events.

For example, row “MDTypeID=1” defines a motion event as a primitiveevent having an absolute value of 3. Row “MDTypeID=22” defines a swipecard read as a primitive event having an absolute value of −1. Row“MDTypeID=23” defines a person detected as a primitive event having anabsolute value of 1. Row “MDTypeID=24” defines tailgating as a compoundevent having an absolute value of 5. Tailgating is defined as a compoundevent consisting of event “23” (person detected), but not event “22”(swipe card read) during a period of 10 seconds (0:10). Note that inthis example, compound events are composed of primitive events usingcombination logic over a period of time. However, this is not the onlyway to represent compound events, and alternative representations, suchas the Allen relations, are also within the scope of the presentinvention.

Rows “MDTypeID=98” and “MDTypeID=99” define a video tip as a primitiveevent having an absolute value of 6, and a stolen plate event as aprimitive event having an absolute value of 10.

Finally, row “MDTypeID=105” defines a network event as a primitive eventhaving an absolute value of 60.

TABLE 3 Events table MDParameter MDEntryID ID MD_Event_DateTimeMD_Event_Duration SrcID Src_Description Src_Locaion . . . . . . . . . .. . . . . . . . . . . 432  6 09/27/2007 7:05:24PM 1:05 1 Camera 1 Lobby433 16 09/27/2007 7:10:18PM 0:01 1 Camera 1 Lobby 434 11 09/27/20078:13:08PM 0:01 9 Card Reader in Server Room Server Room 435 1009/27/2007 8:13:10PM 0:02 4 Camera 34 Server Room 436 10 09/27/20078:13:14PM 0:02 4 Camera 34 Server Room 437 12 09/27/2007 8:13:24PM 0:064 Camera 34 Server Room 438 14 9/27/2007 9:05:00PM 0:26 23  RegisteredOff-campus Student (River St.) 439 15 9/27/2007 9:14:04PM 0:10 2 Camera2 Parking Lot

Table 3 shows an illustrative events table, which corresponds to item118 in FIGS. 1, 2, 4, and 5. The vents table stores the actual primitiveand compound events detected by the present invention. “MDEntryID” is aprimary key that uniquely identifies the event entry, and“MDParameterID” is a foreign key into the Meta-data parameters tablethat defines the type of event that was detected. “MD_Event_DateTime”records the time of the detected event as recorded by the sensorydevice, and “MD_Event_Duration” records the duration of the event asrecorded by the sensory device. Finally, “SrcID”, “SrcDescription”, and“SrcLocation” store information about the source that detected the event(even though this information is already indirectly provided by“MDParameterID”).

For example, Table 3 shows eight illustrative events that were detectedon Sep. 27, 2007. Event “432” of “MDParameterID=6” (corresponding tomotion in the lobby) occurred at 7:05:24 PM, which is within the hoursthat the privacy filter allowed. Event “433” of “MDParameterID=16”(corresponding to a camera in the lobby losing network connection)occurred at 7:10:18 PM. Event “434” of “MDParameterID=11” (correspondingto a swipe card read) occurred at 8:13:08 PM. Event “435” of“MDParameterID=10” (corresponding to the detection of a person) occurredat 8:13:10 PM. Event “436” of “MDParameterID=10” (corresponding to thedetection of a second person) occurred at 8:13:14 PM. Event “437” of“MDParameterID=12” (corresponding to the detection of a tailgatingcompound event) occurred at 8:13:24 PM since no corresponding swipe cardwas detected for 10 seconds when the second person was detected enteringthe server room. Event “438” of “MDParameterID=14” (corresponding to avideo tip received) occurred at 9:05:00 PM. Finally, event “439” of“MDParameterID=15” (corresponding to a stolen plate event) occurred at9:14:04 PM.

This sample of detected events is illustrative of a real scenarioenacted in the laboratory. Note how the two primitive events (secondperson detected, no corresponding swipe card detected) triggered thedetection of a compound event (tailgating). Notice also how the videotip event and stolen plate event were detected. The network managementmodule, which detected that camera 1 lost connection at 7:10:18 PMplaced the network event “MDEntryID=433” into the events database.

Note that the correlation engine would compute the weighted sum of allthese events and generate an alert based on the threshold value (definedbelow in the Rules table). Even though these events may not be related,there is a chance that they are related to one incident. The appropriateauthorities would be notified, and would be given the chance toinvestigate the simultaneous occurrence of multiple suspicious events.

The primitive events may be either generated by sensory devicesthemselves, or by other devices (such as video analytics devices, thenetwork management module, etc.) which take sensory inputs and detectprimitive events in the data. Illustrative primitive events could bemotion detected, gunshot detected, person detected, speed of an object,a camera loses connection, a stolen plate is detected, and similarevents. The sensory devices themselves, the analytics devices, and/oranalytics software running on a general purpose PC, could generate theprimitive events.

In one embodiment of the present invention, a user interface is providedby which a human operator may enter event meta-data. For example, a userinterface is provided for a security officer to monitor one or morecameras. The cameras automatically generate meta-data, as noted above.In addition, the human operator may add meta-data manually. For example,if the human operator observes suspicious activity going on in aparticular camera, the human operator may add meta-data corresponding tosuspicious activity. The human operator may select from a set ofpossible meta-data, as well as add “free-form” meta-data by typing intoa text-entry box. For example, a human operator may transcribe speech inthe video data. The transcribed speech serves as meta-data to the videodata.

TABLE 4 Sources table SrcID 1 4 9 22 23 Src_Description Camera 1 Camera34 Swipe Card Anonymous Registered Reader Student Src_Type IP Camera IPCamera Card Reader Video Tip Video Tip Src_AW_Quality 0.53 0.1 1 0.1 0.5Src_AW_Age 0.54 0.4 0.75 0.1 0.9 Src_AW_Maintenance 0.15 0.15 0.75 0.10.9 Src_AW_Reliability 0.3 0.23 1 0.1 0.9 Resolution 1024 × 768 760 ×640 — — — Dvc_Install 4/30/2007 5/3/2006 5/3/2007 — — Dvc_LifeSpan 5 410 — — Dvc_LastMaint_Date 8/3/2007 8/3/2007 8/3/2007 — —Dvc_Location_Name Lobby Server Room Server Room — — EntranceDvc_Location_Long 42.734534 42.734539 42.734530 — — Dvc_Location_Lat−71.348438 −71.348434 −71.348431 — — Dvc_Angle 45 90 — — —Dvc_MAC_Address 50-1A-01-46 40-8C-7C-A6-F2 25-D6-E4-17 — —Dvc_IP_Address 192.168.1.201 192.168.1.203 192.168.1.49 — — Dvc_Status 11 1 — — Cam_NowImgURL now.jpg cgi-bin/nph-image — — —Cam_ImgStr_RootFolderName cam1 cam34 — VT_Anon VT_Reg

Table 4 shows a sample sources table defining the devices and theirassociated properties (including attribute data) that is used by thecorrelation engine, network management module, and the HSM module. Thesources table is the core table that is used by numerous components ofthe present invention. Most importantly, the sources table storesattribute data for each sensory and other device on the network, whichis used by the correlation engine when assigning weights to the datafrom each sensory device, by the network management module when placingdevices on the physical map and when assigning importance to each devicefor network management events, and by the HSM module to help determinewhich data segments to cascade first and to which hierarchical level.

In the sources table, shown by example in Table 4, “SrcID” is a primarykey used to uniquely identify each device on the network (forsimplicity, only sensory devices are shown in Table 4). “SrcDescription”is a description of each device, such as “IP Camera,” “Swipe CardReader,” “Data Storage Device,” etc. “Src_AW_Quality”, “Src_AW_Age”,“Src_AW_Maintenance”, and “Src_AW_Reliability” are examples of attributedata that may be stored for each source device. As describe previously,the attribute data is used, along with other information, to determinethe relative importance of data from each sensory device. For example,“Src_AW_Quality” is a weight for the quality of the data from thesensory device (video data from higher resolution cameras are weightedhigher), “Src_AW_Age” is a weight corresponding to the age of thesensory device (older sensory devices are weighted lower),“Src_AW_Maintenance” is a weight corresponding to the amount of timeelapsed since the sensory device was last maintained (devices notmaintained in a long time are weighted less), and “Src_AW_Reliability”is a weight corresponding to the reliability of the sensory device (suchas the inverse of its historical false alarm rate). This attribute datais used by the correlation engine for the weights associated with data(w_(i) in Equations 20-22 below). The attribute data shown and describedhere is but an illustrative example of attribute data according to theprinciples of the present invention. Other attribute data may be useddepending on the business needs of an organization using the presentinvention. Other examples of attribute data are described below.

Continuing with the sources table in Table 4, “Resolution” describes theactual resolution of any surveillance cameras (left blank if not asurveillance camera). (Note the difference between “Resolution” which isan actual resolution, versus “Src_AW_Quality” which is a weight that maydepend on the resolution for a surveillance camera.) “Dvc_Install”records the installation date of the device, “Dvc_Lifespan” defines theuseful lifespan of the device, and “Dvc_LastMaint_Date” records the lasttime the device was maintained. (Note that these values are used todetermine the “Src_AW_Age” and “Src_AW_Maintenance” weights.)“Dvc_Location_Name” is a short nickname for the location of the device,“Dvc_Location_Long” stores the longitude coordinate of the physicallocation of the device, “Dvc_Location_Lat” stores the latitudecoordinate of the physical location of the device, while “Dvc_Angle”stores the angle of a surveillance camera (left blank for devices thatdon't have an angle). These values are used by the physical map moduleof the network management module to position the devices on the physicalmap, as well as to shown areas of coverage and areas of darkness (nocoverage). “Dvc_MAC_Address” stores the MAC address of each device,“Dvc_IP_Address” stores the IP address of each device, and “Dvc_Status”is a Boolean flag that stores the network status of each device(1=Online, 0=Offline). These values are used by the network managementmodule to monitor the status of each device on the network. Finally,“Cam_NowImgURL” (stores the URL of the current image for eachsurveillance camera) and “Cam_ImgStr_RootFolderName” (stores the URL ofthe default recording folder for each surveillance camera) are internalvariables used by video recording servers used to record video data.

Different sensory devices, including different cameras, may havedifferent attributes associated with them. Each attribute determines aweight, which could be a constant, or the weight could be a weighingfunction of the attribute. For example, consider a camera 1 that is notdesigned to detect gunshots, but which has a low-quality, integratedmicrophone, and so a gunshot detection component may use the audio todetect loud shots as gunshots. When a motion event is detected on such acamera, it would be assigned a high weight (for example, 0.85 or 85%).On the other hand, if a gunshot was detected on this camera by a gunshotdetection component, the gunshot event would be assigned a low weight(0.05, or 5%) because the camera is known to have a low-qualitymicrophone, and what may have been detected as a gunshot may have justbeen a drop of a metal object. In contrast, gunshot detector 1 may havethe opposite attribute-weight profile, in that motion events from thegunshot detector may be weighted low (say, 0 or 0%) while gunshot eventsmay be weighted high (say, 0.70 or 70%).

Camera 1 may also have an age attribute, indicating the age of thecamera, and an associated weighting function that weights any data fromthe camera with a function that decreases with the age of the camera.The time since the last maintenance of the camera may also serve togenerate a weight. This could be a step-function that is, for example, afunction dropping to zero after 1 year of no maintenance on the camera.The frequency of failure may also serve to weigh any data from thecamera, again using a function that weights network events lower from acamera that has a high frequency of failure. The resolution of thecamera may also serve as attribute data to assign a weight to the data;data from a high-resolution camera would be assigned a higher weightthan data from a lower resolution camera.

Another example of attribute data and associated weights that are tiedto particular meta-data includes weights assigned to meta-dataindicating the number of people in a particular area. This meta-data maybe assigned a high weight (0.80) if it comes from camera 2, which mayhave high resolution, high frame-rate, and other qualities that make itamenable to high reliability for people counting purposes. Contrary, ifthe same meta-data comes from camera 3, which has low resolution, lowframe-rate, or other qualities that make it unreliable when it comes tocounting people, the meta-data may be assigned a low weight (0.40). Inanother example, a 3 Megapixel camera would be weighted higher than aVGA camera for purposes of face recognition or license platerecognition.

A system administrator may enter and customize the attribute data. Asystem administrator would customize the present system by enteringweights that are associated with attribute data. For example, the systemadministrator would select the attribute data that corresponds with eachcamera. One example of administrator-customizable attribute data is thehistorical pattern of a camera being susceptible to being tampered with.A system administrator may identify a low-hanging camera that may beeasily tampered with a lower reliability attribute weight, while ahigh-hanging camera that is difficult to tamper with a higherreliability attribute weight.

The system administrator may customize the attribute data for differentimage qualities. For example, the system administrator would select theweights associated with video data, and the corresponding meta-data,associated with different resolutions of cameras. That is, a higherresolution camera and its associated meta-data would be weighted higherthan a lower resolution camera, and the system administrator wouldselect the relative weights.

The system administrator may set attribute data based on the pastevidence of usefulness of video data coming from each camera. Forexample, a camera that has been useful in the past for detecting,preventing, or prosecuting crimes would be assigned a higher weight bythe system administrator using this user interface. That is, a cameralocated in a high-crime area may be given a higher attribute weight.

Other examples of attribute data include, but are not limited to,reliability of power to the camera; reliability of transmission andbandwidth; susceptibility to noise, interference, and overexposure;weather conditions around the camera; type of camera (day/night, IR,etc.), and so on.

TABLE 5 Rules table Threshold RuleID Nickname MDParamterID ValueContactID MsgTxt 1 Alert 1  6 null 4 Motion in lobby during forbiddenhours 2 Tailgating SR 12 null 1 Tailgating in server room 3 Global Alertnull 61 7 Null 4 Stolen Plate 15 null 2 Stolen plate detected in parkinglot 5 Camera 1 16 null null Camera 1 has lost connection! goes down

Table 5 shows an illustrative Rules table (such as rules table 216 ofFIG. 2) which defines the alerts sent by the alerting engine. Alerts maybe based on single or multiple occurrences of primitive events, singleor multiple occurrences of compound events, or overall system-widecorrelations. For example, an alert may be issued on a single primitiveevent such as motion in the lobby. An alert may also be issued on asingle compound event such as tailgating into the server room. Finally,an alert may also be issued based on overall, system-wide level, such asthe overall system exceeding a threshold value of 61.

In the sample Rules table shown in Table 5, “AlertID” is a primary keyuniquely identifying each rule, “Nickname” provides a nickname for eachrule, “MDParameterID” specifies which event (including primitive orcompound events) that triggers the alert (or null if a system-widealert), “ThresholdValue” specifies a threshold value which triggers analert (for correlated system-wide alerts, or null if an event-basedalert), “ContactID” specifies the group, or individual, that willreceive the alert, or the set of actions that will be triggered by thealert, and “MsgTxt” specifies the text of the message sent on an alert.“ContactID” is a foreign key into another table (not shown) thatspecifies the list of recipients or the list of actions to be performedwhen the alert corresponding to “ContactID” is triggered.

“AlertID=1” corresponds to an alert on a primitive event having anickname “Alert 1” that is triggered on “MDParameterID=6”, which byreference to Table 1 corresponds to motion in Camera 1. “ContactID=4”specifies the individual who will receive the alert, and “MsgTxt”specifies the text of the message sent. (Note that “ThresholdValue” isnull because the alert is on a primitive event, and not a system-widealert.)

“AlertID=2” corresponds to an alert on a compound event having anickname “Tailgating SR” that is triggered on “MDParameterID=12”, whichby reference to Table 1 corresponds to a compound event of tailgating inCamera 34. “ContactID=1” specifies the group of individuals who willreceive the alert, and “MsgTxt” specifies the text of the message sent.(Note that “ThresholdValue” is null because the alert is on a compoundevent, and not a system-wide alert.)

“AlertID=3” corresponds to an alert on a global correlation having anickname “Global Alert” that is triggered when the overall systemreaches a threshold value of 61 (“ThresholdValue=61”). The overallsystem threshold value is calculated by a weighted sum of all eventsentering the system during a given time. The system threshold may becalculated by weighing the events by their associated attribute data, asillustrated below in relation to Equations 20-22. “ContactID=7”specifies the set of actions to be taken when the threshold valueexceeds 61, which could include putting the entire system into adifferent state. (Note that “MDParameterID=null” because this is asystem-wide alert, not an alert on a particular event.)

“AlertID=4” corresponds to an alert on a primitive event having anickname “Stolen Plate” that is triggered on “MDParameterID=15”, whichby reference to Table 1 corresponds to a stolen plate event from thevehicle information module. “ContactID=2” specifies the individual whowill receive the alert, and “MsgTxt” specifies the text of the messagesent. (Note that “ThresholdValue” is null because the alert is on avehicle event, and not a system-wide alert.)

“AlertID=5” corresponds to an alert on a primitive event having anickname “Camera 1 goes down” that is triggered on “MDParameterID=16”,which by reference to Table 1 corresponds to Camera 1 located in thelobby losing network connection. “ContactID=4” specifies the individualwho will receive the alert, and “MsgTxt” specifies the text of themessage sent. (Note that “ThresholdValue” is null because the alert ison a network event, and not a system-wide alert.)

TABLE 6 Video tip meta-data table VideoTip_ID  1 . . .  47 VT_MDEntryID245 438 VT_AnonStatus TRUE . . . FALSE VT_Submit_DateTime 6/12/2007 . .. 9/27/2007 9:05:00PM 3:22:45 PM VT_Email_Addr — . . . joe@sju.eduVT_Phone_Num — . . . 617-455-2233 VT_Name — . . . Joe StevensVT_Location Unknown . . . Parking Lot VT_Submitter_Comment — . . .Suspicious vehicle VT_Reviewer_Comment video too fuzzy . . . Vehicledriving to view erratically VT_IP 192.168.1.45 . . . 192.168.1.243VT_Filename tip23.mp4 . . . abc.avi

Table 6 shows an illustrative Video tip meta-data table which stores themeta-data extracted from video tips. “VideoTip_ID” is a primary key thatuniquely identifies the meta-data associated with each received videotip, while “VT_MDEntryID” is a foreign key into the events table (Table3) which stores the “tip event” generated by the video tip moduleassociated with the video tip. “VT_AnonStatus” is a Boolean value thatindicates whether the video tip is anonymous or not,“VT_Submit_DateTime” specifies the date and time the video tip wassubmitted, “VT_Email_Addr” stores the email address of the source of thevideo tip (if known), “VT_Phone_Num” stores the phone number of thesource of the video tip (if known), “VT_Name” stores the name of thesource of the video tip (if known), “VT_Location” stores the locationthe video tip was taken (if known), “VT_Submitter_Comment” stores anycomments submitted by the tipster, “VT_Reviewer_Comment” stores anycomments entered by the reviewer of the video tip (such as a securityanalyst), “VT_IP” stores the IP address of the device used to submit thevideo tip (if known), and “VT_Filename” stores the filename of the videotip.

Two illustrative video tips are shown in Table 6. The first, with“VideoTip_ID=1” is an anonymous tip since “VT_AnonStatus=TRUE”, whilethe second, with “VideoTip_ID=47” is from a registered student since“VT_Email_Addr=joe@sju.edu” is a valid email address of a registeredstudent.

The first video tip (“VideoTip_ID=1”) has “VT_MDEntryID=245” whichcorresponds to an entry in the events table (this video tip is not shownin Table 3). Since this is an anonymous video tip(“VT_AnonStatus=TRUE”), most of the other fields are blank or unknown.The reviewer added a comment stating that the video tip is too fuzzy toview. Note that the IP address of the computer used to submit the videotip and the filename of the video tip are recorded. Since this is ananonymous video tip, and additionally is hard to view, it is assigned alow attribute weight based on the Sources table (see “SrcID=22” in Table4). This video tip will be largely disregarded by the correlationengine, and will be quickly cascaded to a lower storage hierarchy by theHSM module in order to free up memory on the higher speed devices. Thisvideo tip is likely to be unimportant, and may even be a spurious tipsubmitted by mischievous students, or even adversaries attempting tobreak the system. Accordingly, because the attribute data has resultedin a low weight for this video tip, the present invention is immune toattacks of this kind.

The second video tip shown (“VideoTip_ID=47”) has “VT_MDEntryID=438”which corresponds to an entry in the events table (shown in Table 3). Incontrast to the first video tip, the second video tip is not anonymous(“VT_AnonStatus=FALSE”), and it was submitted on Sep. 27, 2007 at9:05:00 PM from a registered student named Joe Stevens with an emailaddress (joe@sju.edu) and a phone number 617-455-2233. The tipsterincluded meta-data comments stating that a suspicious vehicle wasobserved in the parking lot. The tipster included a short video clip ofthe suspicious vehicle (abc.avi). An authorized reviewer commented thata vehicle was driving erratically in the video clip. Because this tipcomes from a registered student, the Sources table (see “SrcID=23” inTable 4) indicates that it will be weighted heavily by the correlationengine (which may generate an alert that an important video tip wasreceived), and it will be stored longer on the highest hierarchy of datastorage devices for forensic analysis and review.

This example illustrates meta-data and attribute data extracted from avideo tip. The meta-data includes such items as the comments from thesubmitter, the date of submission, and the email address of thesubmitter. The attribute data includes such items as the anonymitystatus of the video tip and the associated weights extracted from theSources table.

TABLE 7 License plate meta-data table LPCaptureListID  1 . . . 456LP_MDEntryID 142 . . . 439 LP_Number F51462 . . . ZEE96 LP_State Florida. . . Michigan LP_ExpDate 05/2009 . . . 07/2010 LP_StolenDate 09/2007 .. . — VIN — . . . — V_Make Ford . . . Chevrolet V_Model Taurus . . .Jeep V_Year 2006 . . . 1999 V_Color Blue . . . Red V_Type 4-door . . .SUV V_Owner_Name — . . . Lisa Smith V_Reg_Date — . . . 01/2006V_Reg_Status — . . . Registered DL_Num — . . . D5069482 DL_State — . . .Michigan DOB — . . . 06/12/1981 SSNum — . . . 052-80-9203 EyeColor — . .. Brown HairColor — . . . Brown Height — . . . 5′8″ Weight — . . . 140Sex — . . . Female Race — . . . Caucasian Warrants_Desc — . . .Outstanding warrant Warrants_IssuedBy — . . . Michigan QueryDate — . . .09/27/2007

Table 7 shows an illustrative License plate meta-data table which storesthe meta-data extracted about vehicles detected in the video.“LPCaptureListlD” is a primary key that uniquely identifies each licenseplate detected and captured in the video data. “LP_MDEntryID” is aforeign key into the Meta-data store table (Table 3) which stores the“vehicle event” generated by the vehicle information modulecorresponding to this license plate. “LP_Number” stores the actuallicense plate detected, “LP_ExpDate” stores the expiration date of thelicense plate, “LP_StolenDate” stores the date the plate was stolen(only relevant for stolen plates, and null if not stolen), “VIN” storesthe vehicle information number used by some vehicles (or null if notknown or not applicable), “V_Make” stores the manufacturer of thevehicle (e.g., Ford), “V_Model” stores the model name of the vehicle(e.g., Taurus), “V_Year” store the year the vehicle was made (e.g.,2007), “V_Color” stores the color of the vehicle (e.g., red), and“V_Type” stores the type of the vehicle (e.g., 4-door).

“V_Owner_Name” stores the name of the registered owner (if known),“V_Reg_Date” stores the registration date of the vehicle (if known), and“V_Reg_Status” stores the registration status of the vehicle(registered, etc.). “DL_Num” stores the driver's license number of theregistered owner (if known), “DL_State” stores the state of the driver'slicense of the registered owner (if known), “DOB” stores the date ofbirth of the registered owner (if known), “SSNum” stores the socialsecurity number of the registered owner (if known), “EyeColor” storesthe eye color of the registered owner (if known), “HairColor” stores thehair color of the registered owner (if known), “Height” stores theheight of the registered owner (if known), “Weight” stores the weight ofthe registered owner (if known), “Sex” stores the sex of the registeredowner (if known), “Race” stores the race of the registered owner (ifknown), “Warrants_Desc” stores any warrant information about theregistered owner (if known and available), “Warrants_IssuedBy” storesthe jurisdiction that issued the warrants (if known and available).

If any of the information is unknown or unavailable, “NULL” is stored.All of this information is retrieved from law enforcement databases(such as state, local, FBI, Interpol databases) by the vehicleinformation module as described previously. The information on thevehicle is populated based on the vehicle's license plate (which may beextracted from the video automatically or entered manually by a humanoperator). Based on the registered owner of the vehicle, informationabout the registered owner (such as warrants, etc.) may be retrievedfrom the law enforcement database(s) by querying based on name. Thepresent invention has been successfully connected to public FBI andInterpol databases, public State of Florida databases on stolen plates,stolen vehicles, etc., as well as private State of Michigan (CLEMIS)database(s). The present invention may be made to work with any existingstate, local, or federal crime enforcement database.

Forensic Analysis

Forensic analysis and event correlation across both space and time maybe performed using the database schemas described here according to theprinciples of the present invention. The events, both primitive andcompound, that are recorded in the events table (Table 3) may be used asindices into the video data. After the events have been stored in theevents table, the events may be used to significantly enhance search andretrieval of the video data. That is, in order to perform a search ofthe video data, the events table may be searched first, and the videodata may be indexed by the events from the events table.

For example, suppose an event was recorded in the events table duringdetection of a person in a particular camera. If at a later time it weredesired to locate all places in the video data where a person wasdetected, a database query would be performed on the events table toretrieve all events where people were detected. The pointers to thevideo data and the indices into the video data would provide a mechanismby which to retrieve the video data that corresponds to thoseoccurrences of people.

FIG. 10 shows a possible set-theoretic explanation of the operation ofthe above historical analysis. Consider the sets of video data V₁, V₂, .. . , V_(i) shown as elements 1002, 1028, and 1030 in FIG. 10respectively. Sets V₁ (element 1002) and V₂ (element 1028) representvideo data from camera 1 and camera 2, respectively, and so on. Each setof video data V_(i) has subsets of video data, for example, subsets fora particular date range, for a particular time range, for a particularevent, etc. For example, video set 1002 has subsets of video dataidentified as elements 1004, 1006, 1008, and 1010 in FIG. 10.

Each set of video data V_(i) has a corresponding set of meta-data M_(i)associated with it. Each element in the set of meta-data M_(i) has anindex, or a pointer, to a corresponding portion of the video data V_(i).For example, meta-data set M₁, shown as element 1012 in FIG. 10, hascorresponding subsets of meta-data, shown as elements 1014, 1016, 1018,and 1020. Each subset of meta-data is indexed, or points to, acorresponding subset of video data. For example, subset 1014 ofmeta-data M₁ is indexed, or points to, subset 1006 of video data V₁ fromcamera 1 (not shown). Note that a one-to-one relationship between videodata and meta-data is illustrated in FIG. 10 for clarity. Therelationship between video-data and meta-data is not restricted to beingone-to-one. The relationship may be one-to-many, many-to-one, as well asmany-to-many.

In addition, sets W_(i) of attribute weight data are weight vectorsassociated with each set of meta-data M_(i) for camera i (not shown).The sets W_(i) of attribute weight data are sets of vectors w_(i,j)which represent weights associated with subsets of the meta-data M_(i).For example, weight vector w_(i,j) represented as element 1024,represents the weights associated with meta-data subset 1016. The weightvectors w_(i,j) may be n-dimensional vectors representing the weights inone of a number of dimensions, each dimension representing a weight in aparticular attribute of the data. For example, a 2-dimentional weight[w₁₁, w₁₂] vector may represent the attribute weights associated withthe reliability of a particular video camera for both motion detectionreliability as well as gunshot detection reliability. One camera mayhave high motion detection reliability and low gunshot detectionreliability, while another camera may have high gunshot detectionreliability and low motion detection reliability. In principle, theattribute weight vectors w_(ij) may be arbitrarily fine-grained withrespect to subsets of the meta-data and subsets of the video data. Inpractice, attribute weight vectors w_(ij) are constant over largesubsets of the meta-data and the video data, and may have largediscontinuities between subsets. For example, gunshot detection devicesmay have a very low motion detection reliability weight, and very highgunshot detection reliability, and vice versa for typical motiondetection cameras.

The set-theoretic described has been shown and described here for easeof understanding and explanation of the present invention. The meta-dataand video data may or may not be stored as sets; the data may be storedin matrices, tables, relational databases, etc. The set description isshown for clarity only. The present invention is not limited to thisparticular mathematical representation, and one of ordinary skill willrecognize numerous alternative and equivalent mathematicalrepresentations of the present invention.

A possible query to retrieve those events in which a person was detectedwould be:SELECT*FROM EVENTS WHERE MDParameterID=10  (1)

Query (1) would retrieve all events where a person was detected. In theset-theoretic notation described above, the query (1) would correspondto:∀x_(j)εV_(i)|M_(i,j)(MDParameterID=10)  (2)

In order to view the video data corresponding to a particular event, apossible follow-on query would be:VIEW EVENT 1  (3)

Similar queries could be used to retrieve other events. For example, inorder to retrieve all tailgating events, a possible query would be:SELECT*FORM EVENTS WHERE MDParameterID=12  (4)

Query (4) would be represented in set-theoretic notation as:∀x_(j)εV_(i)|M_(i,j)(MDParameterID=12)  (5)

To view the first 3 events where tailgating was detected, a possiblequery would be:VIEW EVENT 1, 2, 3  (6)

Another possible query, to search for all video data where a swipe cardwas detected, a possible query would be:SELECT*FROM EVENTS WHERE MDParameterID=11  (7)

Query (7) would be represented in set-theoretic notation as:∀x_(j)εV_(i)|M_(i,j)(MDParameterID=11)  (8)

Similarly, in order to view the video data corresponding to the firsttwo events where a swipe card was detected, a possible query would be:VIEW EVENT 1, 2  (9)

Event searches may be restricted by particular locations or date-ranges.For example, a security analyst may only wish to search a particularcamera, or location, where motion was detected, for example:SELECT*FROM EVENTS WHERE MDParameterID=6 AND SrcID=1  (10)

Query (10) would be represented in set-theoretic notation by restrictingthe search to V₁ (video data from camera 1) as follows:∀x_(j)εV₁|M_(1,j)(MDParameterID=6)  (11)

The security analyst may also restrict searches by date and/or time. Forexample, the security analyst may only wish to search a particular daterange where motion was detected, for example:SELECT*FROM EVENTS WHERE MDParameterID=6 ANDMD_Event_DateTime>=09/26/2007  (12)

Query (12) may be represented in set-theoretic notation as:∀x _(j) εV _(i) |{M _(i,j)(MDParameterID=6)∩M_(i,j)(MD_Event_DateTime>=09/26/2007)}  (13)

Multiple events may also be searched. For example, a security analystmay want to search historical video data for all occurrences where anetwork event was detected or people were detected. A possible query toaccomplish this would be:SELECT*FROM EVENTS WHERE MDParameterID=10 OR MDParameterID=16  (14)

Query (14) may be represented in set theoretic notation as:∀x _(j)εV_(i) |{M _(i,j)(MDParameterID=10)∪M_(i,j)(MDParameterID=16)}  (15)

Any number of combinations and sub-combinations of events may besearched using the query language, including unions and intersections(conjunctions and disjunctions) of events using AND/OR operators, aswell as other logical operators.

Events may also be correlated and analyzed across multiple cameras, ormultiple locations. For example, a security analyst may want to see allevents where motion was detected in a particular lobby, or a stolenplate was detected in a parking lot camera. To perform such a search,the security analyst could search by:SELECT*FROM EVENTS WHERE (MDParameterID=6 AND SrcID=1) OR(MDParameterID=15 AND SrcID=2)  (16)

Query (16) may be interpreted in set-theoretic notation as:∀x _(j) εV ₁ ∪V ₃ |{M _(i,j)(MDParameterID=6)∩M_(2,j)(MDParameterID=15)}  (17)

The security analyst is not required to using a query language. A querylanguage may be used for sophisticated searches. For more basicsearches, a user interface is provided for the security analyst, whichallows the officer to select the meta-data criteria by which to searchby using a visual tool. The user interface automatically generates thequery language and queries the events database for retrieval.

A possible structured query language was shown here. However, thepresent invention is not limited to the query language shown ordescribed here. Any number of query languages are within the scope ofthe present invention, including SQL, IBM BS12, HQL, EJB-QL, Datalog,etc. The query languages described here is not meant to be an exhaustivelist, and are listed here for illustrative purposes only.

When performing queries on meta-data, such as unions and intersections,attribute weights may be recalculated. For example, to recalculate theattribute weights for an intersection of two subsets of meta-data, theattribute weights would be multiplied together, as shown:W(M ₁ ∩M ₂)=W(M ₁)·W(M ₂),  (18)

For example, to calculate the weight associated with two motion eventsoccurring substantially simultaneously, where the first motion event hasa reliability of 90% (0.90), and the second motion event has aprobability of 50% (0.50), the weight associated with both motion eventssubstantially simultaneously is 45% (0.45).

To recalculate the attribute weights for a union of two subsets ofmeta-data, the law of addition of probabilities would be applied, asshown:W(M ₁ ∪M ₂)=W(M ₁)+W(M ₂)−W(M ₁)·W(M ₂)  (19)

For example, to calculate the weight associated with either one of twomotion events occurring substantially simultaneously, where the firstmotion event has a reliability of 90% (0.90), and the second motionevent has a probability of 50% (0.50), the weight associated with eitherone of the events occurring substantially simultaneously is 95% (0.95).

Event Correlation

One embodiment of the present invention allows real-time alerts to beissued based on the present and historical video data, and especiallythe present and historical meta-data (events). In one embodiment of thepresent invention, the correlation engine correlates events, bothpresent and historical, across multiple sensory devices and multiplelocations, and activates via the alert/action engine one or more actionsin response to the correlation exceeding a particular threshold. Aspreviously described, the correlation engine may evaluate various rules,such as “issue an alert to a given destination when a person is detectedin a restricted area during a designated time.” Video analytics devicesare used to extract relevant events from the video data, and are inputinto the correlation engine. Input may also come from other systems,such as other sensory devices (e.g., temperature and pressure probes).Various actions may be taken under certain conditions, and may beactivated by the alert/action engine when a certain set of conditionsare met

In addition to alerting on the occurrence of primitive or compoundevents, the present invention may also alert based on an accumulatedvalue of multiple events across space and time. Equations 20 to 22 showpossible rules that may be evaluated by the correlation engine. Forexample, as shown in Eq. 20, action component a_(l) will be activated ifthe expression on the left-hand side is greater than a predeterminedthreshold τ₁. In Eqs. 20-22, “a” stands for an action, “w” stands forattribute weights, “x” stands for non-video events, and “v” stands forvideo events. Eqs. 20-22 could represent a hierarchy of actions thatwould be activated for different threshold scenarios. Eqs. 20-22 areillustrative of only one embodiment of the present invention, and thepresent invention may be implemented using other equations, otherexpressions.

$\begin{matrix}{{{a_{1}\text{:}\mspace{14mu}{\sum\limits_{i = 1}^{i = N}\;{w_{i} \cdot x_{i}}}} + {\sum\limits_{i = 1}^{m}\;{w_{i} \cdot v_{i}}}} \geq \tau_{1}} & (20) \\{{{{a_{2}\text{:}\mspace{14mu}{\sum\limits_{i = 1}^{i = N}\;{w_{i} \cdot x_{i}}}} + {\sum\limits_{i = 1}^{m}\;{w_{i} \cdot v_{i}}}} \geq \tau_{2}}\ldots} & (21) \\{{{a_{n}\text{:}\mspace{14mu}{\sum\limits_{i = 1}^{i = N}\;{w_{i} \cdot x_{i}}}} + {\sum\limits_{i = 1}^{m}\;{w_{i} \cdot v_{i}}}} \geq \tau_{n}} & (22)\end{matrix}$

Equation 23 shows an example of a calculation for determining weights.The weights “w_(i)” may be a weighted average of attribute data (a),including resolution of the video data (R, “Src_AW_Quality” in Table 4),age of the camera used to capture the video data (A, “Src_AW_Age” inTable 4), time since last maintenance of the camera used to capture thevideo data (TM, “Src_AW_Maintenance” in Table 4), and reliability of thesource of the video data (RS, “Src_AW_Reliability” in Table 4). Notethat a similar expression was used to calculate the importance (Y) ofdata by the HSM module when determining when to cascade data. Otherweighting factors may also be used, and the weighing factors describedhere are illustrative only and are not intended to limit the scope ofthe invention.

$\begin{matrix}{w_{i} = {\sum\limits_{k = 1}^{N}\;{\omega_{k}a_{k}}}} & (23)\end{matrix}$

In equation 23, ω_(k) are relative weights of the attributes (a_(k)),which are themselves weights associated with the data sources. Thepreceding equations are illustrative of but one manner in which thepresent invention may be implemented and are not intended to limit thescope to only these expression(s).

Alternative Embodiments

In one embodiment of the present invention, several user interfaces maybe provided. For example, a user interface may be provided for anadministrator, who can modify various system parameters, such as theprimitive events being detected and recorded, the compound events andtheir definition in terms of primitive events, the attribute data, therules, the thresholds, as well as the action components, alertdestinations, contact lists, and group lists. Another user interface maybe provided for an officer, such as a security guard, to monitor theactivity of the system. For example, a user interface for the securityofficer would allow the officer to monitor alerts system-wide, turn onand off appropriate cameras, and notify authorities. An interface mayalso be provided for an end-user, such as an executive. The interfacefor the end-user allows, for example, the end-user to monitor thosealerts relevant to him or her, as well as to view those cameras andvideo sources he or she has permission to view. Various user interfacesmay be created for various users of the present invention, and thepresent invention is not limited to any particular user interface shownor described here. Other user interface screens, for adding meta-dataand for modifying attribute data, were discussed above.

FIG. 11 shows an example of a hardware architecture 1100 of oneembodiment of the present invention. The present invention may beimplemented using any hardware architecture, of which FIG. 11 isillustrative. A bus 1114 connects the various hardware subsystems. Adisplay 1102 is used to present the operator interface 123 of FIG. 1. AnI/O interface 1104 provides an interface to input devices, such askeyboard and mouse (not shown). A network interface 1105 providesconnectivity to a network, such as an Ethernet network, a Local AreaNetwork (LAN), a Wide Area Network (WAN), an IP network, the Internet,etc. (not shown in FIG. 11). Various sensory devices 1115 may beconnected to the bus 1114. RAM 1106 provides working memory whileexecuting process 1300 of FIG. 13. Program code for execution of process1300 of FIG. 13 may be stored on a hard disk, a removable storage media,a network location, or other location (not shown). CPU 1109 executesprogram code in RAM 1106, and controls the other system components.Privacy rules are stored in privacy database 1107. Meta-data is storedin events database 1108, and attribute data is stored in sourcesdatabase 1109. Hierarchical storage manager 1110 provides an interfaceto one or more storage modules 1112 on which video data is stored. Auditinformation, including data about who, when, and how often someoneaccessed particular video data is stored in audit database 1111. Asstated previously, the separation between event storage, attribute datastorage, and video storage is logical only, and all three storagemodules, or areas, may be implemented on one physical media, as well ason multiple physical media.

Access database 1113 stores access rights and privileges. Access to viewthe video data is only given to those authorized individuals who arelisted in the access database 1113. Access may be restricted based onthe video data, or its associated meta-data. For example, any securityofficer may be able to view the video data taken at night, but onlysecurity officers assigned to investigate a particular case may be givenaccess to the video data where a gunshot was detected.

Access may also be restricted by attribute data. For example, onlycertain high-level security officers may have access to high qualityvideo data from behind a bank teller that may show checks and amounts,whereas any security officer may see the video data from the bank'slobby. Access may also be modulated based on the quality of the videodata. For example, anybody may be able to login and view a VGAresolution view of the lobby of their building, but only the securityofficer can see the mega-pixel resolution video. The access control maybe implemented using an authentication scheme provided by the operatingsystem, such as Microsoft ActiveDirectory™ or LDAP under Linux.

It is to be understood that this is only an illustrative hardwarearchitecture on which the present invention may be implemented, and thepresent invention is not limited to the particular hardware shown ordescribed here. It is also understood that numerous hardware componentshave been omitted for clarity, and that various hardware components maybe added without departing from the spirit and scope of the presentinvention.

FIG. 12 shows another example of a hardware architecture 1200 accordingto another embodiment of the present invention. A network 1220, such asan IP network over Ethernet, interconnects all system components.Digital IP cameras 1215, running integrated servers that serve the videofrom an IP address, may be attached directly to the network. Analoguecameras 1217 may also be attached to the network via analogue encoders1216 that encode the analogue signal and serve the video from an IPaddress. In addition, cameras may be attached to the network via DVRs(Digital Video Recorders) or NVRs (Network Video Recorders), identifiedas element 1211. The video data is recorded and stored on data storageserver 1208. Data storage server 1208 may be used to store the videodata, the meta-data, as well as the attribute data and associatedweights. Data is also archived by data archive server 1213 running theHierarchical Storage Module on enterprise tape library 1214. Data mayalso be duplicated on remote storage 1206 via a dedicated transmissionmedia such as a fiber optic line, or via a public network such as theInternet.

Legacy systems, such as external security systems 1209, may beinterfaced via appropriate normalization engine, as describedpreviously. A central management server 1210 manages the system 1200,provides system administrator, access control, and managementfunctionality. Enterprise master and slave servers 1212 provideadditional common system functionality. Video analytics server 1207provides the video analytics device functionality described above, aswell as providing the interface to search, retrieve, and analyze thevideo data by event stored on data server 1208.

The video, including live feeds, as well as recorded video, may beviewed on smart display matrix 1205. The display matrix includes one ormore monitors, each monitor capable of displaying multiple cameras orvideo views simultaneously. One or more clients are provided to viewlive video data, as well as to analyze historical video data. Supportedclients include PDA 1201 (such as an Apple iPhone®), central client1202, and smart client 1203. A remote client 1204 may be connectedremotely from anywhere on the network or even over the public Internet,due to the open IP backbone of the present invention. FIG. 12 isillustrative of but one hardware architecture compatible with theprinciples of the present invention, and is not intended to limit thescope of the present invention.

FIG. 13 (consisting of FIGS. 13A and 13B) shows a flowchart of a process1300 of a method of video surveillance, storage, and alerting. Theprocess 1300 begins in step 1302, as shown in FIG. 13A. Video data iscaptured from one or more surveillance cameras having attribute data(the attribute data represents the importance of the surveillancecameras), as shown in step 1304. Audio data is captured from one or moreaudio sensory devices having attribute data (the attribute datarepresents importance of the audio sensory devices), as shown in step1306. Primitive video events are detected in the video data byperforming image processing on the video data, as shown in step 1308.Audio events are detected in the audio data by performing audioprocessing on the audio data as shown in step 1310. Video tips arereceived from one or more external sources as shown in step 1312. Tipevents are generated from meta-data and attribute data extracted fromthe video tips (the attribute data represents the importance of thevideo tips), as shown in step 1314. The video data, the audio data, andthe video tips are stored in a hierarchy of two or more data storagedevices, as shown in step 1316. The video data, the audio data, and thevideo tips are cascaded from a first-level storage device to asecond-level storage device based at least on importance of data, asshown in step 1318. (The first-tier device has a higher data accessperformance and a lower storage capacity than the second-tier device.The importance of the data is based on attribute data corresponding tothe source of the video data, the audio data, and the video tips,primitive events detected in the data, time period the data wasrecorded, and time since the data was last accessed.) Process 1300continues on FIG. 13B.

Network events indicative of the network status of all subsystems aregenerated, as shown in step 1324 in FIG. 13B. Vehicle events aregenerated based on information retrieved about a vehicle detected in thevideo data using the detected vehicle's license plate, as shown in step1326. Vehicle events are generated by recognizing a license plate on thedetected vehicle, and generating license plate events containing thedetected license plate. Information is retrieved about the detectedvehicle from a law enforcement database based on the recognized licenseplate. Finally, warrant events corresponding to warrant information fora registered owner of the detected vehicle, wanted person eventscorresponding to wanted person information for the registered owner ofthe detected vehicle, and stolen plate events if the license platecorresponds to a stolen plate are generated. (For ease of presentation,these steps are not shown in FIG. 13B.)

In step 1328, the primitive video events, the audio events, the tipevents, the network events, and the vehicle events (license plateevents, warrant events, wanted person events, and stolen plate events)are normalized. Primitive events are filtered based on a set of privacyrules and business rules, as shown in step 1330. (The set of privacyrules and the set of business rules may be merged into one set ofrules.) Compound events, composed of two or more primitive events, aredetected, as shown in step 1332. [The primitive events include one ormore primitive video events, audio events, tip events, network events,vehicle events (license plate events, warrant events, wanted personevents, and stolen plate events).]

In step 1334, two or more primitive or compound events are correlatedacross both time and space. The primitive events include one or moreprimitive video events from the video analytics devices weighted by theattribute data of the surveillance cameras used to capture the videodata, audio events from the audio analytics devices weighted by theattribute data of the audio devices used to capture the audio data, tipevents from the video tip module weighted by the extracted attributedata of the video tips, network events from the network managementmodule weighted by attribute data of device corresponding to the networkevent, and vehicle events from the vehicle information module weightedby the information retrieved about the vehicle. The compound eventsinclude one or more compound events detected in step 1332 composed oftwo or more primitive events. (Examples of compound events includetailgating, number of people in a designated area, etc. Many examplesare described below.)

In step 1336, one or more rules are evaluated based on the correlationperformed in step 1334. One or more new rules may be generated based onthe correlated events (not shown in FIG. 13B). Finally, one or moreactions (such as alerts to designated individuals) are activated basedon the evaluated rules from step 1336. (Examples of actions includerebooting a camera following a camera freeze, turning on the lights,etc. More examples are described below.)

Primitive Video Events

According to the present invention, various video analytics devices maybe used to generate meta-data, or detect primitive video events, fromthe video data. These video analytics devices may be configured todetect any number of primitive video events. Some illustrative primitivevideo events are listed below. However, the present invention is notlimited to these primitive video events, and various video analyticsdevices may be used to determine one or more primitive video events, andare all within the scope of the present invention.

-   -   1. Presence of intruder in restricted area during restricted        time (excludes false alarms due to pets, wind blowing, trees        moving, etc.)    -   2. Person or vehicle enters a designated area during a        designated time    -   3. Person or vehicle leaves a designated area during a        designated time    -   4. Object left in a restricted area during a designated time    -   5. Object taken from a designated area during a designated time    -   6. Vehicle in a restricted area during a restricted time    -   7. Vehicle driving the wrong way in a designated lane during a        designated time    -   8. Person or vehicle loitering in a designated area during a        designated time    -   9. Speed of motion of an object    -   10. Size of object    -   11. Area of motion of object    -   12. Acceleration of object    -   13. A face detected    -   14. Type of vehicle detected (SUV, car, convertible, etc.)    -   15. License plate of a vehicle    -   16. Speed of a vehicle        Audio Events

The following are illustrative audio events that may be detected byaudio analytics devices:

-   -   1. Gunshot    -   2. Voice level or sound volume    -   3. Certain sound patterns, such as shouts or glass breaking    -   4. Certain key words        Compound Events

Some examples of compound events that may be detected using combinationsand sequences of the primitive events include:

-   -   1. Number of people in designated area    -   2. Detect if more people entered a designated area than left the        designate area    -   3. Detect if a person is too short in designated area    -   4. Detect if a person is too long in designated area    -   5. Number of vehicles in a designated area    -   6. Percent of lane occupied    -   7. Presence of a masked man (person detected but no face        detected)    -   8. Vehicle loitering in a designated area followed by an        intruder in a restricted area during a restricted time    -   9. Multiple people loitering in different restricted locations        during restricted times    -   10. Intruder enters a restricted area during restricted time        followed by a network disturbance (e.g., camera loses connection        to the network)    -   11. Tailgating (unauthorized people following authorized people        into a designated area)    -   12. Possible physical attack on an individual in a secure area        (two people becoming one person)    -   13. Intruder hides himself in a secure area for future damage        (person enters facility but no person leaves facility)    -   14. In two different locations, tailgating, physical attack, or        intruder hiding in secure area    -   15. Count number of cleaning people or authorized personnel        entering a building during a certain time, and identify any        night that number of people entering goes up by more than a        predetermined percentage    -   16. Security person does not show up in a certain period of time        as required    -   17. No person is in designated area when a person is required    -   18. Certain lock is left off door and it is after a certain        time, and no one is in predetermined area (no object left behind        in predetermined area)    -   19. Certain lock is left off multiple gates monitored by        multiple cameras        Other Sensory Devices

Additionally, various sensory devices may be integrated into system 100of FIG. 1 by adding a normalization engine for receiving and processingthe input from the sensory device. Some illustrative sensory devices arelisted below. However, the present invention is not limited to thesesensory devices, and various other sensory devices are within the scopeof the present invention.

-   -   1. Temperature probe    -   2. Pressure probe    -   3. Altitude meter    -   4. Speedometer    -   5. Revolutions per minute meter    -   6. Blood pressure meter    -   7. Heart rate meter    -   8. Chlorine meter    -   9. Radon meter    -   10. Dust particle meter    -   11. Pollution meter    -   12. CO₂ meter    -   13. Bacteria meter    -   14. Water meter    -   15. Electrical meter        Legacy Systems

Interfaces to the following legacy systems or external systems may beprovided by adding an appropriate normalization engine to the system 100of FIG. 1.

-   -   1. Card access (access control systems)    -   2. Personnel systems (retrieve experience levels of personnel        based on recognized face or RFID badge)    -   3. Inventory systems    -   4. Financial systems    -   5. Police dispatch systems    -   6. Currency system    -   7. FBI Most Wanted    -   8. Interpol Wanted Fugitives    -   9. State and Local Law Enforcement Databases—Warrants, Stolen        Vehicles, Stolen Plates, Mug shots    -   10. Light systems    -   11. Access control systems (door locking/unlocking)        Alerts/Actions

As described above, various actions may be performed in response to arule being activated. The alert/action engine may activate one or moreactions under certain conditions defined by the rules. Some illustrativeactions are listed below. However, the present invention is not limitedto these particular actions, and other actions are within the scope ofthe present invention.

-   -   1. Send email to designated person    -   2. Send media-rich alert to Apple iPhone® or other multimedia        hand-held device    -   3. Send SMS to designated phone number    -   4. Connect voice to designated person (IT director, maintenance        person, security)    -   5. Notify authorities/police/fire    -   6. Reboot camera upon failure    -   7. Send alert to public address system    -   8. Send message or picture to police    -   9. Send text message (SMS) to mass list (e.g., all students on a        campus)    -   10. Call designated phone    -   11. Turn lights on or off in a designated area    -   12. Turn thermostat up or down    -   13. Turn camera on or off    -   14. Issue a forced alert (with automatic escalation if no        response)    -   15. Follow a person using Pan-Zoom-Tilt (PTZ) camera    -   16. Follow a person from camera to camera    -   17. Activate electronic locks        Service Components

According to one embodiment of the present invention, service componentsmay be used to integrate human intelligence into the present invention.For example, a service component may provide a user interface for remotesecurity guards who may monitor the video inputs. Some illustrativeexamples of what the security guards could monitor for and detect islisted below. A human operator may detect some events, such as“suspicious behavior,” which may be difficult for a computer to detect.The human operators may also add meta-data for each occurrence of anevent. For example, a security guard may add meta-data to each portionof a video where he or she noticed suspicious activity. The presentinvention is not limited to the examples described here, and is intendedto cover all such service components that may be added to detect variousevents using a human operator.

-   -   1. Detect people going into building but not coming out    -   2. Detect people carrying packages in and not carrying out    -   3. Detect people carrying packages out but not carrying in    -   4. Detect people wearing different clothes    -   5. Detect people acting suspiciously    -   6. Detect people carrying guns    -   7. Detect people tampering with locks    -   8. Detect people being mugged    -   9. Detect a shooting    -   10. Detect people being bullied

The present invention may be implemented using any number of primitiveand compound events, sensory devices, legacy systems, actions, andservice components. Some illustrative components are presented here, butthe present invention is not limited to this list of components. Anadvantage of the present invention is the open architecture, in whichnew components may be added as they are developed.

The components listed above may be reused and combined to createadvanced applications. Using various combinations and sub-combinationsof components, it is possible to assemble many advanced applications.

Real-World Scenarios

The following discussion illustrates just a small selection of advancedapplications that may be created using the above components, anddescribes the occurrences of real shootings that may have been preventedand the assailants apprehended if the present invention was in use.

Consider a scenario corresponding to Virginia Tech, in which 32 peoplewere killed and 24 others were injured. First, a card access is detectedat a secured dormitory entrance while two people walk through theentrance. These two events are compounded and recognized as a tailgatingevent. No alert is issued because the system is in Low-Alert State (thethreshold for an alert has not been exceeded). Next, a gunshot is eitherdetected by a gunshot detector, or a gunshot is reported on campus by astudent (tip). This report puts the system goes into a High Alert State(the absolute value of a gunshot event is high in the Meta-data typestable). This event automatically triggers a warning email to the entirecampus community.

Following this event, a card access at a secured dormitory entrance isdetected again, while two people walk through the entrance. These twoevents are compounded and recognized as a tailgating event. An alert isautomatically issued to an operator based on the tailgating compoundevent because the system is in High Alert State. The operator'sattention is drawn to the particular camera that corresponds to thetailgating alert, and he or she instantly looks at the tailgating videoand sees that the tailgater is carrying a suspicious object (e.g., couldbe a gun). The operator immediately triggers an alert email/SMS messageto residents of that dormitory to stay inside their rooms and lock theirdoors. Thus, the operation of the present system at Virginia Tech couldhave saved lives. The killer walked around campus for two hours, andtailgated into a secure facility over 2 hours after the first gunshotswere reported.

Consider another scenario corresponding to a stalker. On day 1, a carloiters outside a dormitory for an hour. The loitering event isdetected, stored, and indexed, but no alert is generated. On day 2, thecar again loiters outside the dormitory for an hour. The loitering eventis detected, stored, and indexed, but no alert is generated. A woman inthe building reports that she is being stalked. The security guardqueries for multiple instances of loitering cars over the past two days.The security guard identifies the vehicle of the stalker (and writesdown license plate number), and confirms the stalker's identity with thewoman. The security guard runs the license plate through law enforcementdatabases as previously described and checks for outstanding warrants,which come back as negative. The security guard then creates a new ruleto generate an alert when vehicles loiter outside that particularbuilding. On day 3, the car again loiters outside the dormitory. Analert is generated by the system based on the new rule and sent to thesecurity guard. The security guard positively identifies the car as thesame car as in the previous occasions. Finally, the security guarddispatches the police to stop the vehicle and inquire into the driver. Apossible rape, stalking incident, violence, or altercation may have beenprevented.

Consider another a scenario at a construction site. A truck drives up toa construction site at 2 AM. The video and corresponding event is storedbecause a rule was previously defined to detect vehicles in restrictedareas during certain hours, but no alert is generated (since it could bea patrol officer). Five minutes later, the network management moduledetects that a camera monitoring the construction site has lostconnection, and generates a network management event. The correlationengine correlates the two events (vehicle in restricted area) and acamera in the same location losing connection, and an alert is generatedto a security guard showing the two anomalous events (the truck in therestricted area and the camera failure) on a map. The security guard isgiven an option to either monitor other cameras in the area inreal-time, dispatch an officer to the site, and/or raise the alert levelin the area of the construction site, so that other events whichnormally would not have triggered an alert now would.

Several examples of illustrative scenarios in which the presentinvention could be applied were described here. However, as will beimmediately recognized by one of ordinary skill, the present inventionis not limited to these particular scenarios. The present inventioncould be used to help prevent and fight crime as well as ensure safetyprocedures are followed.

In one embodiment, a system administrator may set the rules. The systemadministrator may hold an ordered, procedural workshop with the usersand key people of the organization using the present invention todetermine which primitive events to detect, which compound events todetect, what weighing criteria (attribute data) to assign to devices,and what alerting thresholds to use, as well as who should receive whichalerts.

In another embodiment, the rules may be heuristically updated. Forexample, the rules may be learned based on past occurrences. In oneembodiment, a learning component may be added which can recognizemissing rules. If an alert was not issued when it should have been, anadministrator of the system may note this, and a new rule may beautomatically generated. For example, if too many alerts were beinggenerated for motion in the parking lot, the weights associated with thetime would be adjusted.

More Alternative Embodiments

Various embodiments of the present include a method, a system, and anapparatus of video surveillance having network management, hierarchicaldata storage, a video tip module, and a vehicle information module.

One embodiment of the present invention is a video surveillance,storage, and alerting system (“the system”), including the followingcomponents. One or more surveillance cameras for capturing video datahaving attribute data (the attribute data represents importance of thesurveillance cameras). One or more video analytics devices, adapted toprocess the video data from one or more of the surveillance cameras andto detect primitive video events in the video data. One or more audiosensory devices for capturing audio data having attribute data (theattribute data represents importance of the audio sensory devices). Oneor more audio analytics devices adapted to process the audio data fromone or more of the audio sensory devices and to detect audio events inthe audio data. A video tip module for receiving video tips from one ormore external sources, adapted to extract meta-data and attribute datafrom the video tips and to generate tip events based on the extractedmeta-data and attribute data, the attribute data representing theimportance of the video tips. (A “video tip” is a tip consisting of avideo clip, an audio clip, a still image, or other multimediainformation which can be submitted from a cell phone, or any portablecamera.) A hierarchy of two or more data storage devices for storing thevideo data from the surveillance cameras, the audio data from the audiosensory devices, and the video tips from the video tip module. (Thehierarchy of data storage devices is connected to the surveillancecameras, the audio sensory devices, and the video tip module via anetwork.) A hierarchical storage manager for managing storage andcascade of the video data, the audio data, and the video tips in thehierarchy of data storage devices based on the corresponding attributedata. A network management module for monitoring network status of thesurveillance cameras, the audio sensory devices, and the data storagedevices, the network management module adapted to generate networkevents reflective of the network status of all subsystems. A vehicleinformation module for retrieving information about a vehicle detectedin the video data based on the detected vehicle's license plate, andadapted to generate vehicle events based on the information retrievedabout the vehicle. A correlation engine for correlating two or moreprimitive events, the primitive events including primitive video eventsfrom the video analytics devices weighted by the attribute data of thesurveillance cameras used to capture the video data, audio events fromthe audio analytics devices weighted by the attribute data of the audiodevices used to capture the audio data, tip events from the video tipmodule weighted by the extracted attribute data, network events from thenetwork management module weighted by attribute data of devicescorresponding to the network event, and vehicle events from the vehicleinformation module weighted by the information retrieved about thevehicle. And an alert/action engine for generating one or more alertsand performing one or more actions based on the correlation performed bythe correlation engine.

Another embodiment of the present invention is the system describedabove that also includes a normalization engine for normalizing theprimitive events from the video analytics devices, the audio analyticsdevices, the video tip module, the network management module, and thevehicle information module. Yet another embodiment of the presentinvention is the system described above where the correlation engineincludes a privacy filter for filtering out primitive events normalizedby the normalization engine based on a set of privacy rules, and abusiness filter for filtering out primitive events normalized by thenormalization engine based on a set of business rules. Yet anotherembodiment of the present invention is the system described above wherethe correlation engine also includes a compound event detection modulefor detecting compound events composed of two or more primitive events.Yet another embodiment of the present invention is the system describedabove where the correlation engine also includes a first eventcorrelation module for correlating the primitive events and the compoundevents across time, a second event correlation module for correlatingthe primitive events and the compound events across space, and a rulesengine for evaluating one or more rules based on the correlationperformed by the first event correlation module and the second eventcorrelation module.

Yet another embodiment of the present invention is the system describedabove that also includes a learning engine for generating one or morenew rules based on the primitive events correlated by the correlationengine and the alerts generated by the alert engine. Another embodimentof the present invention is the system described above where the networkmanagement module includes a topological map module for constructing atopological map of the network, where the topological map includes iconsfor the surveillance cameras, the audio sensory devices, and the datastorage devices, and where the icons are connected by lines representinga backbone of the network. Yet another embodiment of the presentinvention is the system described above where the network managementmodule also includes a physical map module for constructing a physicalmap of the network, where the physical map includes icons correspondingto physical locations of the surveillance cameras, the audio sensorydevices, and the data storage devices, and where the physical mapincludes at least a street map view and a satellite map view. Yetanother embodiment of the present invention is the system describedabove where the icons corresponding to the physical locations of thesurveillance cameras have plumes indicating line-of-sight of thesurveillance cameras.

Yet another embodiment of the present invention is the system describedabove where the icons and their associated plumes indicate a networkstate as well as a change of network state of the surveillance camerasas determined by the network management module, and where the physicalmap shows areas of coverage as well as dark areas indicative of thenetwork state of the surveillance cameras. Yet another embodiment of thepresent invention is the system described above where the iconscorresponding to the physical locations of the audio sensory deviceshave concentric circles indicating an area of coverage of the audiosensory devices. Another embodiment of the present invention is thesystem described above where the hierarchical storage manager queries asources table database to extract attribute data about sensory devicesused to capture data being cascaded. Yet another embodiment of thepresent invention is the system described above where the hierarchy ofdata storage devices includes at least a first-tier device and asecond-tier device, the first-tier device having a higher data accessperformance and a lower storage capacity than the second-tier device,and where the hierarchical storage manager cascades the video data fromthe first-tier device to the second-tier device based at least onimportance of the video data. Yet another embodiment of the presentinvention is the system described above where the hierarchical storagemanager includes a rules module for determining storage locations forsegments of video data based on a set of rules based on the importanceof the video data, and a rules update module for updating the set ofrules for segments of video data based on historical access patterns.Yet another embodiment of the present invention is the system describedabove where the importance of the video data is calculated based on theprimitive events detected in the video data, time period the video datawas recorded, and time since the video data was last accessed. Yetanother embodiment of the present invention is the system describedabove where the importance of the video data is calculated as a weightedaverage of attributes of the video data, where the attributes includeresolution of the video data, age of the surveillance camera used tocapture the video data, time since the surveillance camera was lastmaintained, location of the surveillance camera used to capture thevideo data, and primitive events detected in the video data. Yet anotherembodiment of the present invention is the system described above wherethe first-tier device is a disk array and the second-tier device is atape array.

Another embodiment of the present invention is the system describedabove where the vehicle information module includes an automatic licenseplate recognition module for recognizing a license plate on the vehicle,where the vehicle information module generates license plate eventscorresponding to the recognized license plate, and where the vehicleinformation module retrieves information from a law enforcement databasebased on the recognized license plate. Yet another embodiment of thepresent invention is the system described above where the vehicleinformation module generates warrant events corresponding to warrantinformation for a registered owner of the vehicle, and where thecorrelation engine correlates warrant events from the vehicleinformation module with other primitive events.

Yet another embodiment of the present invention is the system describedabove where the vehicle information module generates wanted personevents corresponding to wanted person information for a registered ownerof the vehicle, and where the correlation engine correlates wantedperson events from the vehicle information module with other primitiveevents. Yet another embodiment of the present invention is the systemdescribed above where the vehicle information module generates stolenplate events if the license plate corresponds to a stolen plate, andwhere the correlation engine correlates stolen plate events from thevehicle information module with other primitive events. Yet anotherembodiment of the present invention is the system described above wherethe vehicle information module returns pictures of a registered owner ofthe vehicle, and where the alerting engine sends the picture of theregistered owner of the vehicle to a designated destination if a wantedperson event is triggered for the registered owner of the vehicle.

Other embodiments of the present invention include the methodscorresponding to the systems describe above.

While the methods disclosed herein have been described and shown withreference to particular operations performed in a particular order, itwill be understood that these operations may be combined, sub-divided,or re-ordered to form equivalent methods without departing from theteachings of the present invention. Accordingly, unless specificallyindicated herein, the order and grouping of the operations is not alimitation of the present invention.

While the invention has been particularly shown and described withreference to embodiments thereof, it will be understood by those skilledin the art that various other changes in the form and details may bemade without departing from the spirit and scope of the invention.

The invention claimed is:
 1. A safety system to ensure safety proceduresare followed, comprising: one or more sensors for capturing sensorydata; a data storage device for storing the sensory data from the one ormore sensors; one or more processors, operatively coupled to the one ormore sensors; and one or more memories, operatively coupled to the oneor more processors, the one or more memories comprising program codewhich when executed causes the one or more processors to: capturesensory data from the one or more sensors; process the sensory data fromthe one or more sensors, weighted by attribute data representinginformation about the sensors used to capture the sensory data, todetect primitive events in the sensory data; correlate two or moreprimitive events to determine one or more correlated events; and performone or more actions to ensure safety procedures are followed based onthe correlation performed in the correlating step.
 2. The safety systemof claim 1, wherein the one or more sensors are selected from the groupconsisting of surveillance camera, surveillance microphone, temperatureprobe, pressure probe, speedometer, chlorine meter, radon meter, dustparticle meter, CO₂ meter, water meter, electrical meter, blood pressuremeter, and heart rate meter.
 3. The safety system of claim 1, furthercomprising program code to: detect compound events composed of two ormore primitive events.
 4. The safety system of claim 3, furthercomprising program code to: time correlate the primitive events and thecompound events across time; space correlate the primitive events andthe compound events across space; and evaluate one or more rules basedon the correlation performed in the time correlating step and the spacecorrelating step.
 5. The safety system of claim 1, further comprisingprogram code to: generate one or more new rules based on the primitiveevents correlated in the correlating step and the actions performed inthe action step.
 6. The safety system of claim 1, further comprisingprogram code to: receive tip data from one or more external sources;determine attribute data for the tip data, the attribute datarepresenting the reliability of the source of the tip data; and generatetip events based on the tip data and the attribute data.
 7. The safetysystem of claim 1, further comprising program code to: manage storageand cascade of the sensory data in a hierarchy of data storage devicesbased on the attribute data corresponding to a source of the sensorydata.
 8. The safety system of claim 7, wherein the hierarchy of datastorage devices includes at least a first-tier device and a second-tierdevice, the first-tier device having a higher data access performanceand a lower storage capacity than the second-tier device.
 9. The safetysystem of claim 8, further comprising program code to: determine storagelocations for segments of sensory data based on a set of rules based onan importance of the sensory data.
 10. The safety system of claim 1,further comprising program code to: monitor network status of the one ormore sensors; and generate network events reflective of the networkstatus of the one or more sensors.
 11. The safety system of claim 1,further comprising program code to: retrieve information about a vehicledetected in the sensory data based on the detected vehicle's licenseplate; and generate vehicle events based on the information retrievedabout the vehicle.
 12. A safety system to ensure safety procedures arefollowed, comprising: one or more sensors for capturing sensory data; adata storage device for storing the sensory data from the one or moresensors; one or more data processing units, adapted to process thesensory data from one or more of the sensors, weighted by attribute datarepresenting information about the sensors used to capture the sensorydata, and to detect primitive events in the sensory data; a correlationengine for correlating two or more primitive events from the dataprocessing units to determine one or more correlated events; and anaction engine for performing one or more actions to ensure safetyprocedures are followed based on the correlation performed by thecorrelation engine.
 13. A safety method, comprising the steps of:capturing sensory data from one or more sensors; storing the sensorydata from the one or more sensors in a data storage device; processingthe sensory data from the sensors, weighted by attribute datarepresenting information about the sensors used to capture the sensorydata, to detect primitive events in the sensory data using a computerprocessor; correlating two or more primitive events to determine one ormore correlated events using the computer processor; and performing oneor more actions to ensure safety procedures are followed based on thecorrelation performed in the correlating step.
 14. The safety method ofclaim 13, further comprising: detecting compound events composed of twoor more primitive events.
 15. The safety method of claim 14, furthercomprising: time correlating the primitive events and the compoundevents across time; space correlating the primitive events and thecompound events across space; and evaluating one or more rules based onthe correlation performed in the time correlating step and the spacecorrelating step.
 16. The safety method of claim 13, further comprising:receiving tip data from one or more external sources; determiningattribute data for the tip data, the attribute data representing thereliability of the source of the tip data; and generating tip eventsbased on the tip data and the attribute data.
 17. The safety method ofclaim 13, further comprising: managing storage and cascade of thesensory data in a hierarchy of data storage devices based on theattribute data corresponding to a source of the sensory data.
 18. Thesafety method of claim 17, further comprising: determining storagelocations for segments of sensory data based on a set of rules based onan importance of the sensory data.
 19. The safety method of claim 13,further comprising: monitoring network status of the one or moresensors; and generating network events reflective of the network statusof the one or more sensors.
 20. The safety method of claim 13, furthercomprising: retrieving information about a vehicle detected in thesensory data based on the detected vehicle's license plate; andgenerating vehicle events based on the information retrieved about thevehicle.